acceptodds
Under review as a conference paper at ICLR 2027

Track2View: 4D-Consistent Camera-Controlled Video Generation via Paired 3D Point Tracks

Abstract

Re-rendering an existing video from a novel camera viewpoint requires the output to follow the prescribed camera trajectory while preserving the appearance and dynamics of the original scene across every frame. Existing methods either inject camera parameters and leave source-to-target correspondence implicit, or render estimated geometry frame by frame; neither provides an explicit, temporally continuous link between source and target pixels. We propose Track2View, which conditions a video diffusion transformer on paired 3D point tracks: sparse trajectories of scene points projected into both the source and target camera views. These tracks provide explicit spatiotemporal correspondences that are temporally continuous by construction, encoding what content should appear where and when. At the core of Track2View is a dual-view track conditioner that transfers visual context from source to target view through parameter-free geometric operations and learned temporal aggregation, ensuring generalization to arbitrary camera trajectories without memorizing specific motions. We further introduce a data curation pipeline that extracts one-to-one track correspondences by running a 3D point tracker on temporally concatenated multi-camera view pairs. On a 400-video benchmark spanning static and dynamic scenes, Track2View achieves state-of-the-art results across visual quality, view synchronization, and camera accuracy, reducing rotation error by 20–39% and translation error by 58–66% relative to leading baselines.

Primary area:
generative models
Keywords:
Video Generation, Object Tracking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GZ8bOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies camera-controlled video re-rendering: generating a frame-synchronized target video along a specified camera trajectory while preserving the source scene’s appearance and dynamics. Track2View conditions a pretrained video diffusion transformer on paired 3D point tracks. Its dual-view conditioner transfers source-video features through deterministic bilinear sampling and scattering, combined with learned temporal aggregation and view-specific depth encoding. The paper also introduces a reverse-and-concatenate pipeline for constructing paired track annotations. Evaluation includes a 400-video real-world benchmark and additional controlled experiments on Kubric-4D.

Critical strengths
  1. Paired 3D tracks provide explicit geometric feature transfer and temporal aggregation for synchronized camera-controlled video generation.
  2. Cross-dataset evaluation provides convincing evidence of generalization: the model trained on synthetic MultiCamVideo is tested on real videos from RealEstate10K and MiraData.
Critical weaknesses
  1. Table 2 does not consistently favor the ALL (1152) configuration: Compared with First only (576), the ALL (1152) configuration shows only small gains and even increases TransErr.
  2. Each ablation comparison is evaluated on only one benchmark. Repeating the ablations on both RealCam-Vid and Kubric-4D Moving-20 would better establish the robustness of the design choices across real and synthetic settings.
Questions for authors

Could you compare with a monocular-compatible 4D Gaussian Splatting baseline [1] that reconstructs a dynamic scene from source frames and poses, then renders the target poses? Comparing training costs and evaluation metrics with explicit reconstruction methods would better clarify the benefits of generative re-rendering.

[1] Wu G, Yi T, Fang J, et al. 4d gaussian splatting for real-time dynamic scene rendering[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 20310-20320.

4 comments
AuthorsOpenReview ↗

Response to Reviewe GZ8b (Part1)

We thank the reviewer for the careful reading, valuable feedback, and the constructive suggestions. Following them, we added experiments and analysis. Our point-by-point responses follow.

W1 — Table 2 does not favor All. With the 8-layer model on both benchmarks (Table W1), the result is driven by the accuracy and the coverage of the tracks, not by their number.

We repeated the ablation with the final 8-layer model on RealCam-Vid and report the same comparison on Kubric-4D Moving-20 with ground-truth tracks (Table W1). We do not claim that using all tracks is best. Our view, which the two benchmarks support, is that generation quality depends on the accuracy and the coverage of the conditioning tracks rather than on their number.

Coverage: First only has as many tracks as Rand. ½ but places all queries in a single frame, and it is the weakest setting in camera accuracy on RealCam-Vid and in PSNR on Kubric-4D, whereas halving or quartering the tracks at random while keeping both query frames costs little (within 0.06 CLIP-V on RealCam-Vid and 0.35 dB on Kubric-4D). Accuracy: App. E shows that a confidence-selected subset is more accurate than the dense pool and that, with estimated tracks, adding more tracks lowers PSNR. All is our default because it covers both query frames without a selection heuristic. We will replace Table 2 by Table W1 and rewrite the corresponding text in terms of accuracy and coverage rather than track density.

Table W1 — Ablation on the query strategy, 8-layer model (RealCam-Vid and Kubric-4D Moving-20)

Query strategyRealCam-Vid CLIP-V↑RealCam-Vid Mat.Pix.(K)↑RealCam-Vid RotErr(°)↓RealCam-Vid TransErr↓Kubric-4D PSNR↑Kubric-4D SSIM↑Kubric-4D LPIPS↓
First only92.530.6792.311.8221.830.6770.205
Rand. ½92.580.6801.851.2422.160.6790.194
Rand. ¼92.520.6762.011.4222.070.6770.196
All93.220.6951.550.81822.040.6750.196

W2 — Both ablations are now reported on both benchmarks: temporal depth (Table W2) and query strategy (Table W1); the trends agree.

We agree and have repeated the temporal-depth ablation of Table 3 on the second benchmark. Table W2 reports the number of temporal aggregation layers on RealCam-Vid (Table 3) and on Kubric-4D Moving-20 side by side, with all query tracks in both cases. The trend is the same on the two datasets: two and four layers are close, and the gain comes from eight layers. Table W1 above does the same for the query-strategy ablation of Table 2: the two benchmarks agree that the differences between strategies are small and that the weakest setting is the one with the poorest coverage (First only), not the one with the fewest tracks. We will add both tables to the appendix.

Table W2 — Ablation on temporal aggregation depth (All queries)

Temporal layersRealCam-Vid CLIP-V↑RealCam-Vid Mat.Pix.(K)↑RealCam-Vid RotErr↓RealCam-Vid TransErr↓Kubric-4D PSNR↑Kubric-4D SSIM↑Kubric-4D LPIPS↓
292.800.6852.241.68121.000.6200.233
492.520.6791.971.21421.070.6310.231
893.220.6951.550.81822.040.6750.196
AuthorsOpenReview ↗

Response to Reviewe GZ8b (Part2)

Q1 — 4D Gaussian Splatting baseline run on Moving-20 with ground-truth poses and point-cloud initialisation (Table Q1): 19.10 vs. 22.16 dB PSNR, 90.7% vs. 100% coverage, 503 s of per-scene optimisation vs. none.

We ran the comparison with the official 4D Gaussian Splatting code [1] on the Kubric-4D Moving-20 benchmark of Tab. 6, where both the input poses and the ground-truth target views exist. For each of the 20 scenes, 4DGS is optimised from scratch with its default D-NeRF configuration on the 81 source frames, with ground-truth camera poses and ground-truth point-cloud initialisation, and then renders the target views of Tab. 6; Track2View is evaluated as in Tab. 6. Because 4DGS can only reconstruct what the source trajectory observed, we also report its coverage and its PSNR restricted to the covered pixels (Table Q1).

4DGS reaches 19.10 / 0.560 / 0.314 (PSNR / SSIM / LPIPS) with 90.7% coverage and 21.88 dB on the covered pixels, after 503 s of optimisation per scene on one H100; Track2View reaches 22.16 / 0.675 / 0.196 with 100% coverage, 22.76 dB on the same pixels and no per-scene optimisation. The gap grows with scene dynamics: splitting the 20 scenes at the median number of dynamic objects, 4DGS obtains 20.84 vs. 22.88 dB on the less dynamic half and 17.35 vs. 21.44 dB on the more dynamic half. 4DGS is ahead only on scenes with small camera and object motion and falls to 12–14 dB under large motion; since its train-view PSNR is 43 dB, the failure is extrapolation beyond what the source trajectory observed, not under-fitting. Generative re-rendering therefore brings three things a per-scene reconstruction cannot: no optimisation at test time, generation of the regions the source never observed, and stable dynamic content. We will add Table Q1 to the appendix.

Table Q1 — 4D Gaussian Splatting vs. Track2View on Kubric-4D Moving-20 (20 scenes, target frames of Tab. 6)

MethodPer-scene optimisationPSNR↑SSIM↑LPIPS↓CoveragePSNR on covered pixels
4DGS [1]503 s / scene (H100)19.100.5600.31490.7%21.88
Track2View (ours)none22.160.6750.196100%22.76

PSNR by scene dynamics (the 20 scenes split at the median number of dynamic objects)

Scenes4DGS [1]Track2View (ours)
Fewer dynamic objects20.8422.88
More dynamic objects17.3521.44

References [1] 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering. CVPR 2024.

We hope these results address the reviewer’s concerns and would be glad to clarify any remaining point during the discussion period.

The authors’ response addressed my concerns. I consider this a solid paper with strong theoretical and empirical support. I have a minor reservation about the paper’s novelty. My overall score remains 3.

We sincerely thank the reviewer for the positive feedback and for confirming that our response addressed the concerns.

Reviewer_i1asOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Track2View, a camera-controlled video re-generation method that conditions a video diffusion model on paired 3D point tracks to establish explicit spatiotemporal correspondences between source and target views. The authors also introduce a data curation method based on temporal concatenation of multi-view videos. Experiments demonstrate improvements in generation quality, view consistency, and camera accuracy over existing methods.

Critical strengths
  1. Using paired 3D point tracks for explicit spatiotemporal correspondence is a reasonable approach to camera-controlled video generation compared to relying solely on camera poses.

  2. The method achieves competitive results on RealCam-Vid and Kubric-4D, particularly in camera control accuracy.

Critical weaknesses
  1. The authors construct paired tracks by temporally concatenating multi-view videos. I am somewhat confused about this approach. During training, the input target tracks are extracted from the target video, whereas during inference, tracks are estimated only from the source video and reprojected into the target view. Could this lead to a train-test mismatch?

  2. The current evaluation mainly relies on CLIP similarity, the number of feature matches, and camera pose errors, which are insufficient to directly demonstrate the geometric consistency of dynamic objects across views and time. A direct evaluation of 4D consistency is missing. In addition, The proposed track-based approach appears effective in normal scenes without occlusion, but how does it perform in more challenging scenarios involving large viewpoint changes and object occlusions? The authors should include evaluations on these scenarios.

Questions for authors
  1. Can the authors compare jointly extracted paired tracks with source-only reprojected tracks, or otherwise analyze the train-test distribution gap?

  2. Can the authors directly evaluate the cross-view and temporal geometric consistency of dynamic objects?

  3. Can the authors provide evaluations under large viewpoint changes and object occlusions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer i1as (Part1)

We thank the reviewer for the careful reading, valuable feedback, and the constructive suggestions. Following them, we added experiments and analysis. Our point-by-point responses follow.

W1 & Q1 — Train/test mismatch of the target tracks: measured directly, the effect is below 0.2 dB (Table W1Q1).

We agree that the target tracks are obtained differently: in training they are tracked on the concatenated source–target pair, whereas at inference the 3D tracks of the source video are projected to the target view. We measured the effect directly: on 10 unseen MultiCamVideo camera pairs we generated the target video twice with the same model and target camera, changing only the source of the target tracks.

Table W1Q1 shows that the model is not sensitive to how the target tracks were obtained. The gap is small by design: during training the tracks are queried only on source-video frames, never on target frames, exactly as at inference, so the model always sees tracks that originate in the source view. All real-data results in the paper (Tables 1–3) already use the inference-style pipeline. We will add this analysis to the appendix.

Table W1Q1 — Same model, seed and target camera; only the source of the target tracks changes (10 unseen MultiCamVideo pairs)

Target tracksPSNR↑SSIM↑LPIPS↓
Training-style (tracked on the source–target pair)21.330.6460.190
Inference-style (source tracks projected to the target view)21.170.6450.195
AuthorsOpenReview ↗

Response to Reviewer i1as (Part2)

W2 & Q2 — Direct evaluation of 4D consistency: real data has no ground-truth geometry, so we (i) separate dynamic from static scenes on RealCam-Vid (Tables W2Q2a, W2Q2b) and (ii) measure consistency against ground-truth geometry on Kubric-4D, also on the moving objects alone (Table W2Q2c).

On real data (RealCam-Vid), where there is no ground-truth 4D geometry, we separate the two kinds of content: Tables W2Q2a and W2Q2b report Table 1 separately for the 200 static scenes (RealEstate10K) and the 200 dynamic scenes (MiraData), with the same per-video results as Table 1. Track2View is the best method in almost every column of both subsets, and its advantage over the baselines is larger on the dynamic scenes than on the static ones.

To isolate the generation of the moving objects, we use Kubric-4D Moving-20, where the target video is rendered from the ground-truth 4D geometry (Appendix C) and PSNR against it measures cross-view and temporal consistency directly. We render the ground-truth instance labels into the target camera and report PSNR separately on the pixels of the moving objects, of all objects and of the background (Table W2Q2c). Track2View is the most accurate method on the moving-object region at all three horizons, as well as on all objects and on the background. Together with Table 7, where paired tracks beat warping with exact depth by 3.02 dB, this indicates that the paired tracks carry the motion of the objects across views, not only the camera motion. We will add Table W2Q2c to the appendix.

Table W2Q2a — Table 1 on the 200 static scenes (RealEstate10K)

MethodFramesCLIP-T↑CLIP-F↑CLIP-V↑Mat.Pix.(K)↑RotErr↓TransErr↓
Trajectory Attention2528.5699.1095.480.9871.030.104
Track2View2528.8999.4696.611.0501.120.058
TrajectoryCrafter4928.5499.2494.840.8021.670.468
DaS4928.3298.9393.810.7253.362.723
Track2View4928.8799.5295.790.8731.070.205
Gen3C8128.3499.3293.710.6163.072.708
ReCamMaster8128.2399.4694.010.5663.291.944
Track2View8128.6999.5494.560.6991.210.535

Table W2Q2b — Table 1 on the 200 dynamic scenes (MiraData)

MethodFramesCLIP-T↑CLIP-F↑CLIP-V↑Mat.Pix.(K)↑RotErr↓TransErr↓
Trajectory Attention2528.7198.5993.250.9392.060.305
Track2View2529.2299.1095.001.0901.360.111
TrajectoryCrafter4929.0698.7592.340.7892.611.164
DaS4928.4797.9989.940.6403.825.049
Track2View4929.2799.1993.640.8791.530.357
Gen3C8128.7698.7691.220.6715.374.737
ReCamMaster8128.7399.1991.860.5924.122.252
Track2View8129.0999.2291.890.6921.891.101

Same per-video results and protocol as Table 1; the two subsets add up to the Table 1 numbers. FID is not reported on the 200-video subsets (set-level metric with a sample-size bias).

Table W2Q2c — PSNR↑ by image region on Kubric-4D Moving-20

MethodFramesAll pixelsMoving objectsAll objectsBackground
Trajectory Attention2521.5717.4917.8823.18
Track2View2524.3920.5020.7925.72
TrajectoryCrafter4918.6516.3716.3919.51
DaS4918.4315.1815.2419.67
Track2View4923.2619.7019.9624.49
Gen3C8121.8819.3019.4822.86
ReCamMaster8116.7614.0213.9518.26
Track2View8123.1919.6419.9024.42
AuthorsOpenReview ↗

Response to Reviewer i1as (Part3)

W2 & Q3 — Large viewpoint changes and occlusions: the Kubric-4D Moving-20 benchmark is built to be extreme, and on its harder half, as on the hardest 30% of RealCam-Vid, Track2View stays the most accurate and degrades less than the strongest baselines (Tables W2Q3a, W2Q3b).

Large viewpoint changes and occlusions are exactly what the Kubric-4D Moving-20 benchmark of Appendix C was built to stress: its median source–target relative rotation is 57.7° (max. 130.9°), the median baseline 13.4 m, every scene contains an object rotating by ≥90°, and 30.6% of the source track-time is occluded or out of view.

To show how each method degrades under extreme conditions, we split the 20 scenes at the median of two difficulty measures computed from input metadata only (fraction of occluded source track-time; source–target relative rotation) and report the harder half next to the full set (Table W2Q3a). On the more occluded half Track2View loses 0.7–0.9 dB relative to the full set, whereas Trajectory Attention and Gen3C lose about 1.5 dB; Gen3C is competitive only on the easier scenes. On real data we report the RealCam-Vid subset with the largest camera and object motion (top 30%, 120 videos) next to the full set (Table W2Q3b): Track2View remains the most accurate method on the hard subset, and its camera errors grow the least (RotErr +0.2–0.5° vs. +0.45–0.9° for the baselines; TransErr essentially unchanged). We will add both difficulty splits to the appendix.

Table W2Q3a — Moving-20, harder half by occlusion and by relative rotation

MethodFramesPSNR↑ AllPSNR↑ More occl.PSNR↑ Larger rot.SSIM↑ AllSSIM↑ More occl.SSIM↑ Larger rot.LPIPS↓ AllLPIPS↓ More occl.LPIPS↓ Larger rot.
Trajectory Attention2521.5720.0520.990.6760.6160.6520.2660.3150.304
Track2View2523.3922.5422.270.7120.6870.6880.1660.1960.197
TrajectoryCrafter4918.6518.1117.480.5200.5170.4850.3340.3570.397
DaS4918.4317.4418.590.4940.4670.5020.4070.4380.416
Track2View4922.2321.5321.340.6810.6590.6610.1920.2190.226
Gen3C8121.8820.3320.960.6760.6430.6420.1870.2210.228
ReCamMaster8116.7616.1516.820.4740.4760.5040.4760.5140.524
Track2View8122.1621.4421.290.6790.6560.6580.1940.2210.228

Table W2Q3b — RealCam-Vid, hardest 30% by camera and object motion (120 videos)

MethodFramesCLIP-V↑ AllCLIP-V↑ Hard 30%Mat.Pix.(K)↑ AllMat.Pix.(K)↑ Hard 30%RotErr↓ AllRotErr↓ Hard 30%TransErr↓ AllTransErr↓ Hard 30%
Trajectory Attention2594.3793.410.9630.8871.552.000.2030.266
Track2View2595.8195.181.0701.0051.241.450.0850.089
TrajectoryCrafter4993.5992.720.7960.7322.142.650.8151.016
DaS4991.8890.770.6820.5853.594.063.8802.669
Track2View4994.7193.890.8760.7931.311.670.2800.309
Gen3C8192.4791.410.6440.5984.215.123.7152.697
ReCamMaster8192.9391.720.5790.4993.704.332.0962.123
Track2View8193.2292.020.6950.6031.552.010.8180.809

We hope these results address the reviewer’s concerns and would be glad to clarify any remaining point during the discussion period.

Reviewer_2axnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Track2View re-renders an existing video along a new camera trajectory by conditioning a Wan2.1-1.3B video DiT on paired 3D point tracks, i.e., the same tracked 3D points projected into both the source and target views. A dual-view track conditioner samples source features along the tracks, aggregates them over time with a transformer, and scatters them to the source and target token grids. Paired training tracks are obtained by running SpatialTrackerV2 on a reversed source video concatenated with the target video. On 400 RealCam-Vid videos, the method outperforms five baselines in visual quality, view synchronization and camera accuracy, with additional ablations on Kubric-4D.

Critical strengths
  1. A clean idea. Using identity-preserving tracks instead of camera embeddings or per-frame warped RGB gives an explicit and temporally continuous source-to-target link. The data trick (reverse the source video and concatenate it with the target so one tracker run gives paired tracks) is simple and reusable, and the released track annotations would be useful to others.
  2. Broad evaluation. Three metric families on a real benchmark, VBench, a GT synthetic benchmark, and controlled ablations. Table 7 shows a 3 dB gain over GT-depth warping on Moving-20, where warping loses much of the coverage.
Critical weaknesses
  1. The camera-accuracy numbers, the main claim, are hard to interpret. RotErr/TransErr are computed only on videos GLOMAP can register, and outliers are dropped from TransErr (App. A), but the number of videos kept per method is never reported. Methods with bad outputs may benefit from having them dropped. In addition, at 25/49 frames Track2View is conditioned on the full 81-frame source, and its temporal aggregator can pass information from later frames, while baselines only see 25/49 frames. The gains are largest exactly at 25 frames.

  2. Several comparisons and ablations do not support the claims. On Kubric-4D (Table 6), Track2View is trained on the Kubric training split with GT tracks, while baselines are zero-shot. The learned cross-attention variant K4 scores below the zero-shot baselines, which suggests a weak reimplementation rather than an advantage of parameter-free transfer. Table 2 differences are at noise level and run with 2 layers, not the final 8. With estimated tracks (Table 12), the gain over GT-depth warping mostly disappears. The promised static/dynamic split is never reported, so "4D-consistent" is not directly tested.

Questions for authors
  1. How many of the 400 videos are registered and kept for each method and frame length in Table 1? Please also report camera errors on the intersection of videos registered by all methods.
  2. Please re-run the 25/49-frame results using only the first 25/49 source frames.
  3. Please report Table 1 separately for the 200 static and 200 dynamic scenes.
  4. How are target trajectories defined on RealCam-Vid, and how is the SpatialTrackerV2 scale aligned to them?
  5. How was K4 validated (training curves, longer training)? Please also add Table 2 with 8 layers.
  6. The related work could discuss recent geometry-guided video re-rendering methods such as EX-4D (https://arxiv.org/abs/2506.05554), Light-X (https://arxiv.org/abs/2512.05115) and Vista4D (https://arxiv.org/abs/2604.21915). These methods also re-render monocular videos along new camera trajectories using explicit 4D geometry (meshes or point clouds), which is the main alternative the paper argues against. A discussion of how paired tracks compare to these approaches would help position the contribution.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 2axn (Part1)

We thank the reviewer for the careful reading, valuable feedback, and the constructive suggestions. Following them, we added experiments and analysis. Our point-by-point responses follow.

W1 & Q1 — Camera-accuracy protocol: registration counts while using GLOMAP and common-subset errors added (Table W1Q1); the ranking is unchanged.

We follow the camera-accuracy protocol of CameraCtrl [1] and ReCamMaster [2], and agree that the number of videos behind each number should be reported and that a common subset is the fairer comparison. Table W1Q1 reports, for every method and frame count, (i) how many of the 400 test videos GLOMAP registers, (ii) the fraction of frames registered within them, and (iii) RotErr / TransErr on the method’s own set (= Tab. 1), on the videos registered by all methods, and on the videos in which every frame is registered by all methods.

Registration is nearly complete for every method: 400/400 videos at 25 and 49 frames; 397 / 392 / 400 at 81 frames for Gen3C / ReCamMaster / Track2View; 99.6–100% of frames kept. The common subsets are therefore large (400 / 400 / 389 videos; 398 / 391 / 386 for the all-frames intersection). On both common subsets the numbers move by hundredths and the ranking is unchanged at every frame count.

We will include Table W1Q1 in the revision.

Table W1Q1 (camera accuracy with registration statistics)

FramesMethodVideos keptFrames keptOwn set (Tab. 1) Rot / TransRegistered by all Rot / TransAll frames by all Rot / Trans
25Trajectory Attention (fair V2V)400/400100.0%1.55 / 0.2031.55 / 0.2031.51 / 0.204
25Track2View400/40099.9%1.24 / 0.0851.24 / 0.0851.19 / 0.085
49TrajectoryCrafter (fair V2V)400/40099.9%2.14 / 0.8152.14 / 0.8152.05 / 0.811
49DaS (I2V)400/40099.6%3.59 / 3.8803.59 / 3.8803.51 / 3.941
49Track2View400/400100.0%1.30 / 0.2811.30 / 0.2811.24 / 0.279
81Gen3C397/400100.0%4.21 / 3.7154.17 / 3.7224.06 / 3.750
81ReCamMaster392/40099.8%3.70 / 2.0963.70 / 2.1093.64 / 2.111
81Track2View400/40099.9%1.55 / 0.8181.54 / 0.7791.47 / 0.785

W1 & Q2 — 25/49-frame rows re-run with the source truncated to the first 25 / 49 frames (Table Q2): appearance moves toward the baselines, camera accuracy stays the lowest.

The reviewer is right that the 25- and 49-frame rows of Tab. 1 conditioned Track2View on the full 81-frame source, so part of the gain at those lengths came from frames the baselines never see. We re-ran both settings with the source video and tracks truncated to the first 25 / 49 frames (Table Q2).

The appearance metrics move toward the baselines, as expected for a model whose base video generator (Wan2.1) is pretrained to produce 81-frame clips and whose track conditioner is trained in that same regime, so 25- and 49-frame generation lies outside its training distribution; Track2View nevertheless remains ahead of Trajectory Attention and TrajectoryCrafter in CLIP-T, CLIP-F, CLIP-V and Mat.Pix at both lengths. Camera accuracy, our main claim, is not affected by the truncation: RotErr and TransErr remain the lowest at 25 and 49 frames (0.78° / 0.076 and 0.96° / 0.172), because the camera signal comes from the paired tracks of the frames the model actually sees, the same frames the baselines receive.

We will add the truncated-source rows to Tab. 1 and state the conditioning explicitly in the revision.

Table Q2 (Tab. 1, 25/49-frame block with the truncated-source rows)

MethodFramesCLIP-T↑CLIP-F↑CLIP-V↑Mat.Pix.(K)↑RotErr(°)↓TransErr↓
Trajectory Attention2528.6398.8494.370.9631.550.203
Track2View (Ours), 81-frame source2529.0699.2895.811.0701.240.085
Track2View (Ours), 25-frame source2528.9099.2695.531.0570.780.076
TrajectoryCrafter4928.8098.9993.590.7962.140.815
DaS*4928.3998.4691.880.6823.593.88
Track2View (Ours), 81-frame source4929.0799.3594.710.8761.310.280
Track2View (Ours), 49-frame source4928.8499.3294.380.8640.960.172
AuthorsOpenReview ↗

Response to Reviewer 2axn (Part2)

W2a — Table 6 vs. zero-shot baselines: a zero-shot Track2View row is added and the training data of every method is marked (Table W2a).

Table 6 reports our Kubric-trained model. To separate the effect of track conditioning from that of in-domain training, Table W2a adds Track2View evaluated zero-shot: the RealCam-Vid model of Table 1 applied to Moving-20 without any Kubric training, conditioned on 1,000 ground-truth tracks. It also marks whether each method used Kubric data in training; Gen3C did, so it is not a zero-shot baseline. Without any Kubric training, Track2View outperforms Trajectory Attention (25 frames), TrajectoryCrafter and DaS (49) and ReCamMaster (81) in PSNR and LPIPS, with a lower SSIM than Trajectory Attention; our Kubric-trained model exceeds Gen3C on all three metrics. We will add the zero-shot row and the training-data column to Table 6.

The remaining points of W2 (K4 training, the Table 2 differences, the static/dynamic split) are answered under Q5 and Q3.

Table W2a — Table 6 with zero-shot Track2View (Kubric-4D Moving-20; PSNR / SSIM / LPIPS)

MethodFramesZero-shotPSNR↑SSIM↑LPIPS↓
Trajectory Attention25✓21.570.6760.266
Track2View (ours)25✓22.140.6410.227
Track2View (ours), Kubric-trained25✗24.390.7530.146
TrajectoryCrafter49✓18.650.5200.334
DaS*49✓18.430.4940.407
Track2View (ours)49✓21.140.6130.258
Track2View (ours), Kubric-trained49✗23.260.7240.170
ReCamMaster81✓16.760.4740.476
Track2View (ours)81✓21.070.6100.261
Gen3C81✗21.880.6760.187
Track2View (ours), Kubric-trained81✗23.190.7210.171

W2b — Estimated tracks vs. GT-depth warping, compared under the same geometry source (Table W2b): paired tracks win under both ground-truth and estimated geometry.

The two experiments behind this comparison take different conditions as input: in Table 12 our model receives estimated tracks, whereas the warping baseline of Table 7 receives ground-truth depth. We therefore evaluated dense warping with estimated source depth as well (Table W2b). Under both ground-truth and estimated geometry, paired tracks outperform dense warping on every metric: dense warping passes every depth error into the conditioning frames, while sparse tracks only displace a few hundred points. Table 12 shows the same effect within our method, where more estimated tracks give worse results (19.34 dB with 1,000 tracks vs. 18.92 dB with 24,960). We will add the estimated-depth warping row to Table 12 and state the geometry source of every row.

Table W2b — Paired tracks vs. dense warping under ground-truth and estimated geometry (Kubric-4D Moving-20, 81 frames, 1,000 tracks; same backbone, data and schedule)

ConditioningGeometry sourcePSNR↑SSIM↑LPIPS↓
Dense warping (K2)GT source depth19.140.5600.308
Paired tracks (ours)GT tracks21.960.6720.209
Dense warping (K2)estimated source depth18.490.5050.331
Paired tracks (ours)estimated tracks19.340.5260.285
AuthorsOpenReview ↗

Response to Reviewer 2axn (Part3)

Q3 — Table 1 split into the 200 static and 200 dynamic scenes (Tables Q3a, Q3b): Track2View is best in almost every column of both, with a larger margin on the dynamic scenes.

Tables Q3a and Q3b use the same per-video results as Table 1. Dynamic scenes are harder for every method (lower CLIP-V, higher camera errors), and Track2View’s advantage over the baselines is larger there than on the static scenes. We will add both tables to the appendix.

Table Q3a — Static scenes (200 videos, RealEstate10K)

MethodFramesCLIP-T↑CLIP-F↑CLIP-V↑Mat.Pix.(K)↑RotErr↓TransErr↓
Trajectory Attention2528.5699.1095.480.9871.030.104
Track2View2528.8999.4696.611.0501.120.058
TrajectoryCrafter4928.5499.2494.840.8021.670.468
DaS4928.3298.9393.810.7253.362.723
Track2View4928.8799.5295.790.8731.070.205
Gen3C8128.3499.3293.710.6163.072.708
ReCamMaster8128.2399.4694.010.5663.291.944
Track2View8128.6999.5494.560.6991.210.535

Table Q3b — Dynamic scenes (200 videos, MiraData)

MethodFramesCLIP-T↑CLIP-F↑CLIP-V↑Mat.Pix.(K)↑RotErr↓TransErr↓
Trajectory Attention2528.7198.5993.250.9392.060.305
Track2View2529.2299.1095.001.0901.360.111
TrajectoryCrafter4929.0698.7592.340.7892.611.164
DaS4928.4797.9989.940.6403.825.049
Track2View4929.2799.1993.640.8791.530.357
Gen3C8128.7698.7691.220.6715.374.737
ReCamMaster8128.7399.1991.860.5924.122.252
Track2View8129.0999.2291.890.6921.891.101

Q4 — Target trajectories follow ReCamMaster’s definition, anchored at the source camera of frame 0; SpatialTrackerV2 is aligned to Depth Anything 3 metric depth with one per-video scale.

Target trajectories: we follow ReCamMaster [2]. The target camera is anchored to the source camera at frame 0 and then follows the given trajectory, Pitgt=P0srcT0−1Ti\mathbf{P}^{\text{tgt}}_i = \mathbf{P}^{\text{src}}_0 \mathbf{T}_0^{-1}\mathbf{T}_i, where P0src\mathbf{P}^{\text{src}}_0 is the source pose at frame 0 and Ti\mathbf{T}_i the ii-th pose of the ReCamMaster trajectory in meters, so P0tgt=P0src\mathbf{P}^{\text{tgt}}_0 = \mathbf{P}^{\text{src}}_0. Scale: SpatialTrackerV2 is defined only up to a global scale, so we align its depth to the metric depth of Depth Anything 3 [3] with a single per-video factor; the metric trajectory is then applied directly. We will add these details to Appendix A.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.