acceptodds
Under review as a conference paper at ICLR 2027

CorrRetime: Correspondence-Guided Object-wise Space-Time Retiming

Abstract

Recent video diffusion methods have explored temporal control, but typically apply a single global transformation to the entire scene, making it difficult to independently control the temporal evolution of multiple objects and the camera/background within the same video. We introduce CorrRetime, a controllable video diffusion framework for fine-grained object-wise temporal remapping with independent camera/background retiming. Our key idea is to represent this control through explicit spatiotemporal correspondences that specify, for each output frame, both the source-time state of each object and its target location under the retimed camera trajectory. This enables multiple foreground objects to independently accelerate, decelerate, pause, resume, or reverse while preserving their underlying source motion, while the camera/background can be independently retimed along the original camera path according to a separate temporal schedule. To enable supervised training and systematic evaluation of this task, we construct a large-scale synthetic dataset with independently parameterized object and camera temporal schedules, spanning diverse multi-object retiming patterns, camera trajectories, and occlusion relationships. We evaluate CorrRetime on both synthetic and real-world videos, demonstrating accurate object-wise and camera/background temporal control while outperforming adapted baselines in retiming accuracy and visual quality.

Primary area:
generative models
Keywords:
video generation, video editing, generative ai

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AhYYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CorrRetime, a video-to-video diffusion framework for object-wise temporal retiming under independently controlled object and camera timelines. The key idea is to represent each retiming operation through explicit source-to-target spatiotemporal correspondences, which associate an object state at a specific source time and location with its desired target time and location after camera reprojection. These correspondences, together with the source video, retimed background reference, and object masks, are injected into a pretrained Wan2.1 video diffusion model. The method is trained on synthetic source-target video pairs with known depth, masks, camera poses, and retiming schedules. Evaluation is conducted on a synthetic benchmark with ground-truth retimed videos and on 62 real-world videos using an anchor-based reconstruction protocol, FVD, and a user study. The paper reports improvements over adapted TrajectoryCrafter and ReCamMaster baselines on both synthetic and real-world evaluations.

Critical strengths
  1. The paper gives a concrete formulation for object-wise retiming by linking each target location to a specific source-time state. This control signal is well matched to the task and can be injected directly into the video diffusion backbone.
  2. The synthetic setup provides paired ground-truth retimed videos, which makes direct quantitative evaluation possible. The adapted TrajectoryCrafter and ReCamMaster variants use the same backbone and training budget.
Critical weaknesses
  1. MotionV2V is discussed as a closely related object-level motion editing method, but absent from the experiments. The absence of this comparison leaves the benefit of explicit source-time correspondence insufficiently isolated from a closely related track-based approach.
  2. The real-world evaluation are computed only at an anchor frame where the object timelines and camera timeline coincide with the observed source frame. These scores provide limited evidence for temporal correctness at the frames where retiming actually changes the source-time state.
Questions for authors

What videos are used as the reference set for the real-world FVD reported in Table 2?

4 comments
AuthorsOpenReview ↗

Response to Reviewer AhYY

We thank the reviewer for the helpful comments.

W1. Comparison with MotionV2V.

MotionV2V and Edit-by-Track edit motion through point trajectories. In particular, Edit-by-Track uses paired source–target 3D tracks together with camera poses, so retiming can in principle be expressed by constructing retimed target trajectories.

In contrast, CorrRetime directly controls the timeline of each object and the camera through independent source-time schedules τₖ(f). Each object can start from a different initial source time τₖ(0) and follow a time-varying playback speed sₖ(f), enabling independent acceleration, deceleration, freezing, and reversal.

CorrRetime automatically constructs source–target spatiotemporal correspondences for each output frame using the object state at the selected source time, estimated depth, and camera geometry. Therefore, it does not require persistent point identities across the entire video or manually constructed retimed point trajectories (General Response, Section 3).

A direct quantitative comparison with MotionV2V and Edit-by-Track was not feasible, as neither method had released inference code or model weights as of November 2026 (the MotionV2V GitHub repository hosts only the project page). We instead compare with Go-with-the-Track (GWTT; Namekata et al., SIGGRAPH 2026), a publicly available track-based method.

Under independent object retiming, CorrRetime achieves a substantially lower timing error than GWTT (4.51 vs. 13.65 frames) and also outperforms it in mPSNR (20.05 vs. 19.84 dB), mSSIM (0.627 vs. 0.584), and mLPIPS (0.126 vs. 0.149).

These results suggest that CorrRetime is particularly effective for independent object-wise temporal control (General Response, Table R7).

W2. Real-world evaluation beyond the anchor frame. We extend our real-video evaluation to all 81 frames, rather than evaluating only the anchor frame (General Response, Tables R1 and R2). Under global retiming, where captured frames provide ground truth throughout the video, CorrRetime achieves a mean frame-matching timing error of 1.61 frames, with 92% of output frames matched within two frames of the requested source time, compared with an average error of 20.68 frames without retiming. CorrRetime also achieves the best object-region mPSNR, mSSIM, and mLPIPS (Table R1). Under independent object retiming, we additionally evaluate object-specific timing using geometry-assisted frame matching and reconstruction quality using the captured ground truth for an object retaining the shared schedule. CorrRetime achieves a timing error of 5.24 frames, compared with 19.23 frames without retiming, while obtaining the best reconstruction quality across all three metrics for the locked object (Table R2).

Q. Reference set for real-world FVD. For Table 2, we compute FVD between each method's generated videos and the 62 original captured real-world videos, using I3D features pretrained on Kinetics-400. The original videos serve as the reference distribution for evaluating visual and temporal distributional similarity. Since retimed videos intentionally differ from the original videos in their motion speed and timing, FVD does not directly measure adherence to the requested temporal schedules. We therefore complement it with the frame-level reconstruction and timing evaluations in Tables R1 and R2.

Thank you for your response, which substantially addresses my concerns. I noticed that the method explicitly supports temporal schedules that move beyond the source timeline and later re-enter via boundary clipping, but such schedules are excluded from synthetic training. This leaves an important claimed behavior unsupported by the training distribution. Could the authors explain the gap?

Thank you for the follow-up and for noting that our response addressed your concerns. We also thank the reviewer for pointing out that our description of rejection sampling in the appendix (Synthetic Dataset Details) could be misinterpreted; we clarify its scope below.

Clipped schedules are included in training. Rejection sampling applies only to piecewise schedules. Constant-speed schedules are retained even when τ(f) leaves the source range, and the target is rendered by holding the boundary pose, consistent with the clipping in Eq. 2. Because each object and the camera independently use either schedule type, 1,609 of the 3,534 training clips containing piecewise schedules also include an out-of-range constant-speed schedule, and a further 1,565 constant-speed training samples contain clipped segments. Holding at the source boundary is therefore directly supervised.

Re-entry after clipping. Schedules that leave the source range and later re-enter it are not directly included in training. Under Eq. 2, however, re-entry produces the same correspondence conditioning as a pause at the boundary frame followed by forward or reverse motion, since the boundary source frame is repeatedly reprojected to the current camera pose during the clipped interval. Such pause-to-motion transitions occur 454 times across 433 piecewise training samples (276 forward, 178 reverse), with independently retimed cameras, which provides a basis for generalizing to re-entry.

Evaluation. Clipped frames. In our real-video global-retiming evaluation (Table R1), 704 of 5,022 frames (14.0%, across 36 of 62 videos) have clipped source times:

FramesCountTiming error ↓Within ±2 frames ↑mPSNR ↑
Clipped7043.9591.3%20.66
Unclipped4,3181.2892.4%19.40

Metrics. Timing error measures the difference between estimated and target source-frame indices. Within ±2 frames denotes the percentage of frames with a timing error of at most two frames. mPSNR measures reconstruction quality over object regions.

Accuracy within two frames and reconstruction quality are maintained on clipped frames. The higher mean timing error (median zero) is driven mainly by two scenes: one with a fast reversed camera, where the background does not follow the requested viewpoint, and one with repetitive motion, where frame-matching timing estimation is ambiguous.

Re-entry. Re-entry is exercised by our anchor-based evaluation: for reversed schedules with anchor 60 (−2×, −3×, −4×), τ(f) exceeds the source range before reversing and re-enters before the anchor frame, which is scored against the captured frame. On these 79 runs, CorrRetime achieves 17.53 dB anchor mPSNR (17.06 dB for TrajectoryCrafter†), compared with 17.71 dB for runs at the same reverse speeds that never leave the source range (anchors 30/45). These results show comparable anchor-frame reconstruction quality for runs with and without re-entry under the evaluated reverse-speed settings. However, since the anchor positions differ and mPSNR is computed over all object regions at the anchor frame, this comparison does not isolate the effect of re-entry or directly measure temporal accuracy during re-entry.

Revision. We will clarify in the appendix that rejection sampling applies only to piecewise schedules, revise the explanation after Eq. 2 to distinguish boundary clipping from re-entry, and add these clipping and re-entry analyses.

Additional analysis: temporal accuracy after re-entry.

To directly evaluate temporal accuracy after re-entry, we analyzed 186 global-retiming runs (62 videos × 3 reverse speeds: −2×, −3×, −4×), all involving source-time clipping followed by re-entry. Unlike our previous anchor-frame evaluation, we evaluated the full sequences using captured-frame ground truth. For each run, we separately analyzed the frames from re-entry up to the anchor frame (11, 7, and 6 frames per run at −2×, −3×, and −4×), clipped frames, and the remaining frames:

SegmentFramesTiming error ↓ (median)Within ±2 frames ↑mPSNR ↑
After re-entry1,4882.12 (1)84.3%17.49
Clipped2,5421.17 (0)97.2%20.66
Remaining11,0361.90 (1)88.4%18.27

Metrics. Timing error measures the difference between estimated and target source-frame indices. Within ±2 frames denotes the percentage of frames with a timing error of at most two frames. mPSNR measures reconstruction quality over object regions.

The mean per-run timing-error difference between post-re-entry and remaining frames is +0.34 frames (95% scene-bootstrap CI [−0.09, +0.85]), suggesting that temporal accuracy remains reasonably stable after re-entry. Although reconstruction quality is lower (mean per-run mPSNR difference: −0.93 dB), this may partly reflect the difficulty of fast reverse motion and cannot be attributed solely to re-entry.

We will include this frame-level analysis, together with per-speed results, in the appendix of the revised manuscript.

Reviewer_4UokOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors introduce CorrRetime, a video-editing method that independently adjusts each object’s timing while allowing separate control over the timing of camera motion. It uses temporal remapping function and scene geometry to guide where objects to appear and a video-generation model to fill in missing content.

Critical strengths

The authors introduce an interesting task formulation: independently controlling the timing of multiple objects and the camera within the same video while preserving the original motion and appearance. The paired synthetic dataset is also a valuable contribution, enabling training and evaluation with ground-truth retimed videos. I thank the authors for including video demos in the supplementary materials, which was very helpful for illustrating the method’s capabilities.

Critical weaknesses

My main concern is that, while the authors identify a unique problem, their proposed solution is itself ill-posed.

The per-object correspondence construction fails to provide sufficient geometric information to handle occlusions, disocclusions, and newly revealed regions, contrary to the authors’ claims. Although the correspondences are computed using 3D geometry, the conditioning ultimately represents pairs of source and target 2D spatiotemporal coordinates. Projecting a source point into the target view tells us where it would appear on the canvus, but not whether it should be visible.

For example, suppose a person looks directly at the source camera, with their nose at the image center (0, 0). The target camera then moves 180° around their head to view it from behind.

The camera orbits horizontally at a fixed distance dd from the head’s center and always looks toward that center. Using camera coordinates with the forward direction along +Z+Z, the 180° transformation from (X,Y,Z)(X, Y, Z) (source coordinates) becomes (−X,Y,2d−Z)(-X, Y, 2d − Z)(target coordinates).

Thus, the nose coordinates are:

Xnosesrc=(0,0,d−r)\mathbf{X}_{\mathrm{nose}}^{\mathrm{src}} = (0,0,d-r)

Xnosetgt=(0,0,d+r)\mathbf{X}_{\mathrm{nose}}^{\mathrm{tgt}} = (0,0,d+r)

Its depth changes, but it still projects to the image center in both views because its horizontal and vertical coordinates are zero. From behind, however, the nose should be hidden by the back of the head. The reprojection is mathematically correct, but a correspondence linking the nose to that target pixel could give the model misleading guidance.

The paper describes using a z-buffer to handle visibility. However, this only compares the available projected points. If those points come from the frontal source frame, they do not include the back of the head, so the z-buffer cannot account for that surface hiding the face.

This concern also extends to the conditioning design.The conditioning encodes time and 2D locations, but does not explicitly include depth or camera poses. Other inputs may help the model infer the viewpoint, but it is unclear whether it handles hidden and newly visible surfaces through geometric guidance or by generating plausible content from its learned prior.

My second concern is the fairness of the baseline. ReCamMaster receives a preprocessed video with the objects already retimed, which may contain holes or other artifacts. Unlike CorrRetime, it does not also receive the original video to help recover missing details. Its weaker results may therefore partly reflect information lost or errors introduced during preprocessing.

Questions for authors

See weakness.

  1. Could the authors clarify how the method handles the large viewpoint change described in my first concern?
  2. The examples appear to use static backgrounds. Does the method assume a static background, or can it handle independently moving background elements? If so, can you provide some demos?
  3. Does the synthetic scenes support continuous animation time? Why are source times rounded to integer frame indices in Equation 2 if continuous animation time is supported? Is this rounding applied during both training and inference, or only when processing real-world videos at inference time?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4Uok (1/2)

We thank the reviewer for the detailed and technically precise comments.

W1 / Q1. Occlusion and large viewpoint changes

We agree with the reviewer's concern. Estimated depth and camera poses are used to construct source–target spatiotemporal correspondences and perform z-buffer visibility filtering, but are not separate inputs to the diffusion model, which receives correspondence guidance. Our z-buffer resolves visibility only among observed source points; newly revealed target surfaces have no source correspondences and must be synthesized using the source information and learned generative prior. We quantify both directions of this visibility issue on synthetic test scenes using ground-truth target depth and masks exclusively for evaluation, not inference. Specifically, we examine (i) source-visible correspondences incorrectly retained in the target and (ii) target-visible surfaces missing from the selected source view.

Source-visible correspondences occluded in the target. We measure source-visible points that pass our z-buffer but are occluded in the target. Excluding a 3-pixel silhouette boundary to reduce rasterization artifacts, the fraction rises from 1.18% at 0–15° relative camera rotation to 4.84% at 30–45° (Table R4). Without the exclusion, it reaches 6.6–10.4%, an upper bound including rasterization noise.

Table R4. Source-visible correspondences passing our z-buffer but occluded in the target view (3-pixel silhouette boundary excluded).

Rotation# Corr.Occluded
0–15°13.8M1.18%
15–30°1.9M2.79%
30–45°0.38M4.84%
45°+0.06M4.54%

In visibility-inconsistent regions, CorrRetime yields LPIPS 0.064 versus 0.096 for TrajectoryCrafter†. Compared with normal regions, LPIPS rises by 35% versus 72%, respectively (Table R5).

Table R5. Reconstruction quality in regions with and without visibility-inconsistent correspondences.

MethodRegionPSNR ↑LPIPS ↓LPIPS increase
TrajectoryCrafter†Normal23.030.056–
TrajectoryCrafter†Inconsistent21.290.096+72%
CorrRetimeNormal23.740.048–
CorrRetimeInconsistent22.360.064+35%

Target-visible surfaces unobserved in the source. We also evaluate the reviewer's reverse case: target object pixels absent from the source view at time τ(i). Excluding a 3-pixel silhouette boundary, they account for 3.74% of target object pixels, rising from 3.07% at 0–15° to 6.92–8.73% at 15–45° relative rotation. Most arise outside the source camera's field of view (2.30% of target object pixels); self-occluded surfaces account for another 0.99%.

Table R5b. Reconstruction quality in target object regions not observed versus observed at the selected source time (3-pixel silhouette boundary excluded).

MethodUnobserved PSNR ↑Observed PSNR ↑Paired ΔPSNRUnobserved LPIPS ↓
TrajectoryCrafter†16.4921.32−4.120.107
CorrRetime18.3521.82−2.030.085

Both methods degrade in unobserved regions, but CorrRetime achieves better PSNR (+1.86 dB) and LPIPS (0.085 vs. 0.107). Its PSNR advantage is larger there than in observed regions (+1.86 vs. +0.50 dB), with a smaller paired PSNR gap (2.03 vs. 4.12 dB). We hypothesize this partly reflects correspondence-based conditioning: unlike conditioning directly on rendered reference videos, it lets diffusion resynthesize target appearances without directly inheriting rendering artifacts. Together with Table R5, these results suggest greater robustness to visibility errors within the tested range, although the missing source surfaces still cannot be recovered from geometry alone.

Because source and target share the same object state, visibility changes arise from relative camera motion, including both rotation and translation. We group results by relative camera rotation to characterize viewpoint-change severity; we do not assume that rotation alone determines visibility. Both poses lie on the original camera trajectory, so evaluated changes are bounded by the motion captured in each clip. The test set includes relative rotations up to approximately 66°, so we evaluate occlusion and disocclusion under substantial viewpoint changes. However, changes as extreme as the reviewer's 180° example would require the original camera to orbit the object and remain outside the evaluated range. We will explicitly state this as a limitation in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 4Uok (2/2)

W2. Fairness of the ReCamMaster† baseline.

We thank the reviewer for raising this concern. ReCamMaster† receives a preprocessed reference video, which may contain holes or artifacts, whereas CorrRetime additionally uses the original source video. We acknowledge that this difference in conditioning information may contribute to ReCamMaster†'s lower performance.

In our adaptation, we preserve ReCamMaster's original single-video conditioning interface rather than introducing an additional input stream that would require architectural modifications. We also control the backbone, training dataset, and training budget to ensure a consistent experimental setup. Thus, our comparison evaluates the adapted methods under their respective conditioning designs, rather than isolating the contribution of correspondence representation alone.

As a complementary comparison, TrajectoryCrafter† receives both the original source video and a rendered reference, yet CorrRetime still achieves higher object-region reconstruction quality. Although this does not fully resolve the conditioning difference with ReCamMaster†, it shows that CorrRetime also outperforms a baseline that uses both the original source video and a rendered reference.

To further address this concern, we have trained an additional ReCamMaster† variant conditioned on both the original source video and the rendered reference video. We report the results in a follow-up comment and will include them in the appendix of the revised manuscript.

Q2. Static background assumption.

CorrRetime does not assume a static background. The background follows the camera's temporal schedule: for each output frame f, the background reference is taken from source frame τ_cam(f). Consequently, dynamic elements such as moving people, vehicles, or foliage that are not designated as controllable objects are naturally included in the background reference and follow the same temporal schedule as the camera.

However, independently retiming background dynamics while maintaining a different camera timeline would require additional scene motion and geometry modeling, which is beyond the scope of our current method. We acknowledge this as a limitation. Moving elements can instead be treated as independently controllable objects by providing their masks and assigning separate temporal schedules.

To further clarify this behavior, we will include qualitative examples featuring dynamic backgrounds in the appendix of the revised manuscript.

Q3. Rounding in Eq. 2.

Our synthetic targets are rendered at continuous animation times, whereas correspondence construction requires source RGB observations, depth maps, and camera poses, which are available only at discrete source frames. We therefore map each requested source time to the nearest available frame (Eq. 2), consistently during training and inference, and for both synthetic and real videos. This introduces a source-time quantization error of at most half a frame.

Importantly, this discretization affects only the geometric correspondence construction, not the target supervision. During training, ground-truth target frames are rendered at their exact continuous animation times. Consequently, the model is supervised to generate intermediate motion states even when the geometric guidance is derived from neighboring discrete source frames. This allows the model to learn temporal interpolation rather than being restricted to repeating the nearest source frame.

Nevertheless, this discretization may limit temporal precision, particularly for rapid motion, which we acknowledge as a limitation of our current correspondence construction.

Update on W2: ReCamMaster† with source-video conditioning.

As promised, we trained a ReCamMaster† variant that receives both the original source video and the rendered reference video (ReCamMaster†+src). Both ReCamMaster† and ReCamMaster†+src are trained separately from the same public ReCamMaster checkpoint, with the same backbone, training data, and training protocol.

Implementation. We concatenate the original source-video latents with the rendered reference latents along the channel dimension, while preserving ReCamMaster†'s temporal concatenation and camera-conditioning architecture. The additional source-video patch embedding is zero-initialized, so the added input does not alter the model's initial behavior, and it is trained alongside the original trainable modules.

Results on the synthetic test set:

MethodPSNR ↑SSIM ↑LPIPS ↓mPSNR ↑mSSIM ↑mLPIPS ↓
ReCamMaster†23.3520.7410.10017.3830.4570.153
ReCamMaster†+src23.8050.7500.10917.0150.4430.162
CorrRetime29.8250.8980.02620.1590.6340.087

Adding the source video slightly improves full-frame PSNR and SSIM, potentially due to additional background context. However, full-frame LPIPS and all three object-region metrics become worse, and a substantial performance gap to CorrRetime remains (6.0 dB in PSNR and 3.1 dB in mPSNR). These results indicate that providing the original source video to ReCamMaster† does not close the performance gap with CorrRetime.

We will include these results in the appendix of the revised manuscript.

Reviewer_aXLPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents CorrRetime, a method for independently retiming different objects in a video and the camera along its original path. It uses depth and camera poses to link points on each object at the selected source time to their projected locations under the retimed camera pose. These correspondences guide a pretrained Wan video model to generate the edited video. The authors build a synthetic dataset for training and evaluation, compare against adapted versions of TrajectoryCrafter and ReCamMaster, and include ablations and a user study on real videos.

Critical strengths
  1. The editing task is useful. Changing the timing of objects and the camera separately means showing object states from viewpoints that were not captured together in the input. The correspondences give the model a clear guide to which source state to use and where it should appear under the retimed camera view.

  2. The comparisons are reasonably well controlled. The adapted baselines use the same backbone, training data, training steps, and geometric inputs. The synthetic data also allows comparison against ground truth. The gains over these baselines, along with the real-video user study, give me confidence that the approach works in the tested settings.

Critical weaknesses
  1. The added value over existing time-control methods is unclear. MotionV2V already shows object-specific time control, so the paper needs to establish what CorrRetime adds. Its main design jointly encodes the source and target time/location of each correspondence. The results show improvements over adapted camera-control baselines, but do not isolate the benefit of this joint encoding. All variants in Table 4 retain it. Comparing joint encoding against separately encoding the two endpoints, while keeping other inputs and training settings fixed, would help clarify its contribution. A comparison with an available time-control method, such as SpaceTimePilot, on a setting both methods support would also strengthen the evaluation.

  2. The real-video reconstruction metrics do not measure timing accuracy across the video. PSNR, SSIM, LPIPS, and their masked versions are computed at an anchor frame where all time mappings match the original video. A method could reconstruct this frame well without following the requested timing elsewhere. The synthetic tests and user study are useful, but more direct evidence is needed on real videos: whether objects follow their schedules, whether editing one object changes others, and where the method fails.

Questions for authors
  1. What specific advantage does CorrRetime offer over MotionV2V and Edit-by-Track for temporal control?

  2. How much does jointly encoding the source and target endpoints help? Can you provide an ablation that encodes them separately while keeping the other inputs and training settings fixed?

  3. Can you compare with SpaceTimePilot on a setting both methods support—for example, a single moving object against a static background, with matched camera paths and separate object/camera time schedules?

  4. How accurately do real-video outputs follow the requested schedules outside the anchor frame? Please provide a direct evaluation of timing accuracy and representative failure cases, including whether editing one object unintentionally changes others.

6 comments
AuthorsOpenReview ↗

Response to Reviewer aXLP (1/2)

We thank the reviewer for the thoughtful questions.

Q1. Advantage over MotionV2V and Edit-by-Track. Both methods edit motion through point trajectories. In particular, Edit-by-Track uses paired source–target 3D tracks together with camera poses, so retiming can in principle be expressed by constructing retimed target trajectories.

In contrast, CorrRetime takes an independent source-time schedule τₖ(f) for each object and the camera, so that the timeline of each can be controlled directly. Each object can start from a different initial source time τₖ(0) and follow a time-varying playback speed sₖ(f), enabling independent acceleration, deceleration, freezing, and reversal.

CorrRetime automatically constructs the source–target spatiotemporal correspondences for each output frame from the object state at the selected source time, depth, and camera geometry. It therefore does not require point identities that persist across the entire video. Further explanation and qualitative examples are provided in General Response, Section 3, and the supplementary material.

A direct quantitative comparison with MotionV2V and Edit-by-Track was not feasible, as neither method had released inference code or model weights as of November 2026 (the MotionV2V GitHub repository hosts only the project page). We instead compare with Go-with-the-Track (GWTT; Namekata et al., SIGGRAPH 2026), a publicly available track-based method.

Under global retiming, GWTT is competitive and attains a slightly higher mPSNR than CorrRetime (19.81 vs. 19.64 dB), while CorrRetime achieves a lower timing error (1.67 vs. 2.22 frames), higher mSSIM (0.631 vs. 0.593), and lower mLPIPS (0.128 vs. 0.150). Under independent object retiming, CorrRetime achieves a substantially lower timing error than GWTT (4.51 vs. 13.65 frames), and also outperforms it in mPSNR (20.05 vs. 19.84 dB), mSSIM (0.627 vs. 0.584), and mLPIPS (0.126 vs. 0.149).

These results suggest that CorrRetime is particularly effective for independent object-wise temporal control (General Response, Table R7).

Q2. Contribution of joint encoding. Following the Table 4 ablation protocol, we compare our joint correspondence encoding with an identity-linked separate-encoding baseline that uses each endpoint's own coordinates and a shared random identity embedding instead of the opposite endpoint's coordinates. The architectures and training settings are matched (64 evaluation samples; 9K training steps). Joint encoding improves object-region mPSNR from 19.781 to 20.028 dB, mSSIM from 0.6023 to 0.6078, and mLPIPS from 0.0905 to 0.0881 (Table R3; General Response, Section 4).

Q3. Comparison with SpaceTimePilot in a shared setting. As suggested, we compare with SpaceTimePilot in two settings it supports (General Response, Section 6 and Table R6): (i) the synthetic setting the reviewer described, with a single moving object, a static background, matched camera paths, and separate object and camera schedules (172 scenes), and (ii) global retiming on all 62 real videos, where captured frames serve as ground truth (248 retimed jobs with 0.25×, 0.5×, and 2× and 4× back-and-forth playback). SpaceTimePilot uses its official checkpoint (50 steps, CFG 5.0), its official camera normalization, the same source frames and camera trajectories as ours, and per-scene captions. CorrRetime achieves higher PSNR than SpaceTimePilot in both the synthetic setting (29.94 vs. 16.96 dB) and on the real retimed videos (25.92 vs. 18.74 dB), and outperforms it on all reported metrics. A substantial gap also remains in 1× reconstruction without retiming (32.67 vs. 20.92 dB), indicating that part of the difference stems from general reconstruction fidelity rather than temporal control alone. Since the two methods use different conditioning mechanisms, we interpret this as a comparison of the complete methods rather than an isolated comparison of their temporal-control representations.

AuthorsOpenReview ↗

Response to Reviewer aXLP (2/2)

Q4. Timing accuracy outside the anchor frame, cross-object effects, and failure cases.

  • Timing accuracy: With real ground truth at every frame (General Response, Table R1), CorrRetime reproduces the requested state within two frames for 92% of output frames, versus about 20-frame errors when the schedule is ignored.
  • Independent objects: Under geometry-assisted timing evaluation, independently retimed objects achieve a lower estimated timing error than without retiming (5.24 vs. 19.23 frames), and CorrRetime achieves the best reconstruction quality across all three metrics for the object that follows the shared schedule (Table R2).
  • Effect on other objects: Retiming the other objects independently changes the quality of the region that keeps the shared schedule by about 0.6 dB PSNR on average (0.56 dB), indicating limited interference in most cases; the effect is largest for closely interacting objects (up to 5.4 dB), as discussed in the failure cases below.
  • Failure cases: We analyze the 1,000 real-video runs of our anchor-based evaluation (Table 2; scored at the anchor frame). CorrRetime achieves better overall reconstruction quality than TrajectoryCrafter† (mPSNR 19.11 vs. 17.60 dB), but its advantage decreases under fast playback and reversal. For forward camera speeds of 0.25×–2×, CorrRetime outperforms TrajectoryCrafter† by 1.5–2.2 dB, whereas at ±4×, it loses in approximately half of the runs. Rapid articulated motion can produce incorrect motion phases: for example, CorrRetime generates a standing pose when the expected pose is crouching or squatting. Fast-moving objects near image boundaries can also be truncated or misplaced, while closely interacting objects may exhibit appearance mixing. These failures suggest that correspondence-based guidance becomes less reliable under large temporal displacements, particularly for rapid articulated motion and complex object interactions.

Thank you for the additional experiments, which address some of my earlier concerns. However, the results still show distorted or disconnected limbs. I am not yet convinced that the current approach can reliably handle fast motion, occlusion, and complex deformations. Despite the improved metrics, I still have concerns about the visual quality. Could the authors clarify whether these failures mainly come from inaccurate correspondences or the model not following the conditioning closely enough?

The method diagrams are also cluttered and hard to follow. Please reorganize the main figures and include more representative videos and close-up comparisons.

For now, I maintain my original rating.

Comments on the paper

8 comments
AuthorsOpenReview ↗

General Response (1/5)

We thank all reviewers for their constructive feedback. The main concerns were (i) real-video evaluation limited to the anchor frame (aXLP, AhYY), (ii) the relation to track-based methods such as MotionV2V (aXLP, AhYY), (iii) the contribution of joint source–target encoding (aXLP), (iv) occlusion handling (4Uok), and (v) comparison with SpaceTimePilot (aXLP). We ran the following new experiments and will upload a revised PDF highlighting all changes.

Summary of new results.

  • Real-video timing beyond the anchor frame (Table R1): scoring all 81 frames of 62 real videos under global retiming, CorrRetime stays within two frames of the requested time for 92% of output frames (1.61 vs. 20.68 frames without retiming) and achieves the best object-region mPSNR, mSSIM, and mLPIPS.
  • Independent object retiming (Table R2): CorrRetime achieves the best reconstruction quality for the object that keeps the shared schedule, while retimed objects reach a much lower estimated timing error than without retiming (5.24 vs. 19.23 frames).
  • Track-based comparison (Table R7): against Go-with-the-Track, CorrRetime reduces the object timing error under independent retiming from 13.65 to 4.51 frames and is better on all quality metrics.
  • Joint encoding (Table R3): joint source–target encoding outperforms identity-linked separate encoding on all three object-region metrics.
  • Occlusion (Tables R4–R5b): visibility errors affect a small fraction of correspondences within our data range (1–5%, increasing with viewpoint change), and CorrRetime degrades less than TrajectoryCrafter† in both affected and unobserved regions.
  • SpaceTimePilot (Table R6): CorrRetime achieves higher PSNR in both the synthetic setting (29.94 vs. 16.96 dB) and on real retimed videos (25.92 vs. 18.74 dB).

1. Real-video timing accuracy beyond the anchor frame (global retiming). Retiming objects and the camera independently creates views that were never captured, so real videos have no ground truth for them. The exception is global retiming: if every object and the camera follow the same schedule τ, output frame i should show the scene exactly as captured at time τ(i). This gives real ground truth at every frame. CorrRetime handles this as a special case of its object-wise control, by giving the camera and every object the same τ.

We use the same schedules as in our original evaluation (slow, fast and reversed playback; 1× excluded), on all 62 real videos, and now score all 81 frames instead of only the anchor. Quality is measured with mPSNR, mSSIM and mLPIPS on object regions, and timing error is the distance between the requested source frame τ(i) and the captured frame that best matches the output (LPIPS nearest neighbor over the whole frame, searched over all 81 captured frames). No retiming outputs the source video unchanged.

Table R1. Global retiming on real videos (62 videos).

MethodTiming err. (frames) ↓mPSNR ↑mSSIM ↑mLPIPS ↓
No retiming20.6813.830.3520.301
ReCamMaster†17.3315.750.4300.242
TrajectoryCrafter†*0.9418.900.6280.150
CorrRetime1.6119.570.6290.129

CorrRetime achieves a timing error of 1.61 frames, staying within two frames for 92% of output frames, compared with 20.68 frames without retiming. It also achieves the best object-region mPSNR, mSSIM, and mLPIPS, improving upon TrajectoryCrafter† by 0.67 dB in mPSNR and 14.0% in mLPIPS. *Under global retiming the source and target cameras coincide, so TrajectoryCrafter†'s reference nearly equals the ground-truth frame, which favors it in timing.

AuthorsOpenReview ↗

General Response (2/5)

2. Independent object retiming. We evaluate whether independently retimed objects follow their requested schedules and whether editing them affects other objects. For each multi-object video, we select one object k to follow the same temporal schedule τ as the camera, while assigning different schedules to all remaining objects. This produces 129 evaluation variants from 57 real videos. We evaluate reconstruction quality and timing accuracy separately:

  • Locked object k (reconstruction quality). Because object k and the camera follow the same schedule, the object's desired appearance at output frame i is directly available in the captured source frame τ(i). We use this frame as ground truth and measure mPSNR, mSSIM, and mLPIPS within the locked object's mask. To avoid comparing pixels affected by changes in occlusion, we exclude regions covered by independently retimed objects in either the ground-truth frame or their requested target positions.
  • Retimed objects (timing accuracy). The desired appearance of a retimed object o at source time τₒ(i), viewed from the camera pose at τ(i), was never captured. Therefore, direct pixel-level ground truth is unavailable. Instead, we estimate the source time represented by the generated object. For each candidate source frame t, we reproject object o into the target camera view using MegaSaM-estimated depth and camera poses. We then compute LPIPS between the generated output and each reprojected candidate within the candidate object's footprint. The source frame t* with the lowest LPIPS is selected, and timing error is defined as |t* − τₒ(i)|. This evaluation covers 174 variant–retimed-object pairs and serves as a proxy for object-specific timing accuracy rather than a direct ground-truth measurement.

Table R2. One object locked to the shared schedule, all other objects retimed independently (quality: 129 variants; timing: 174 variant–object pairs).

MethodIndep. obj. timing err. ↓mPSNR ↑mSSIM ↑mLPIPS ↓
No retiming19.2314.570.3610.281
ReCamMaster†11.6716.550.4470.220
TrajectoryCrafter†4.2919.240.6200.141
CorrRetime5.2419.980.6250.126

Using this approach, independently retimed objects achieve a much lower estimated timing error than without retiming (5.24 vs. 19.23 frames), and CorrRetime achieves the best reconstruction quality across all three metrics for the locked object. TrajectoryCrafter† achieves a slightly lower estimated timing error (4.29 vs. 5.24 frames). Its depth-warped reference is constructed from the requested source time and target camera pose, and our timing evaluation likewise relies on depth-based reprojection. This similarity may favor it in this evaluation, although we cannot rule out that it is genuinely more accurate in timing.

AuthorsOpenReview ↗

General Response (3/5)

3. Relation to track-based methods. MotionV2V and Edit-by-Track primarily control video motion through edited point trajectories. Although these methods can support temporal manipulation, their control is expressed through the motion of tracked points rather than an explicit source-time schedule for each object. Consequently, representing complex object-wise retiming requires constructing target trajectories that correspond to the desired temporal evolution of each object. This can be challenging when tracks are incomplete or unreliable due to occlusion, large temporal displacements, or significant appearance changes.

In contrast, CorrRetime directly specifies which source-time state of each object should appear at each output frame, independently of the camera and other objects. The temporal schedule determines when to sample each object's state, while scene geometry determines where that state should appear under the target camera view. We construct source–target spatiotemporal correspondences for each output frame, linking source coordinates (xₛ, yₛ, tₛ) to target coordinates (xₜ, yₜ, tₜ), without requiring persistent point identities across the video. This decouples explicit temporal control from long-range point tracking, while still relying on geometric estimates and generative priors to handle occlusion and newly revealed regions.

Concretely, this formulation differs from trajectory-based control in two ways:

  • Explicit temporal schedules and non-zero initial states. CorrRetime assigns each object and the camera an independent source-time mapping τₖ(f), with its own initial source time τₖ(0) and potentially varying playback speed sₖ(f), allowing explicit temporal control without manually editing point trajectories. For example, at the first output frame, one object can start from source frame 50 and another from frame 20, while the camera follows a third source time. Each then evolves according to its own temporal schedule, supporting acceleration, deceleration, freezing, and reversal. Qualitative examples are provided in the supplementary material and demo videos.
  • Per-frame correspondence construction. Correspondences are built independently for each output frame from the selected source-time observations, object masks, depth, and camera geometry, directly associating the requested source-time state with its target location. Our method still depends on accurate geometry and cannot geometrically recover surfaces absent from the selected source observations.

Importantly, track-based methods can, in principle, express retiming through appropriately constructed target trajectories. Our distinction is therefore not an inability of trajectory-based methods to represent temporal changes, but a task-specific formulation that directly parameterizes independent temporal schedules and constructs their geometric guidance without long-range point tracking. A direct quantitative comparison with MotionV2V and Edit-by-Track was not feasible, as neither method had released inference code or model weights as of November 2026 (the MotionV2V GitHub repository hosts only the project page).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.