acceptodds
Under review as a conference paper at ICLR 2027

Manifold4D: Denoising on Point Cloud Rendered Manifolds for Video Re-shooting

25 views

Abstract

Video re-shooting re-renders a monocular video of a dynamic scene along a user-specified camera trajectory. A dominant paradigm explicitly provides the target geometry by lifting the source video into a 4D point cloud using per-frame depth, and rasterizing this representation along the target trajectory to obtain point-cloud renders as geometric conditions for trajectory-controlled generation. Because the point cloud render and the source video are both handed to the network as visual conditions, their potentially conflicting cues persist throughout denoising — a tension we refer to as a trust dilemma — which may affect trajectory control, the 3D consistency of dynamic content, or visual quality, particularly on data outside the training distribution. We argue that because a point-cloud render is pixel-aligned with the target view, it does not need to be continuously provided as a conditioning signal throughout denoising. We propose , which injects the render directly into the initial noise of flow matching, so that generation no longer departs from standard Gaussian noise but from a geometry-bearing noise distribution that we term the point cloud rendered manifold, leaving the source video as the only visual condition. The render is thus used exactly once, and the network is never asked to learn how to read it; in subsequent denoising steps the model can focus on the source video. On our benchmarks, Manifold4D attains the best camera-control accuracy on every metric, lowering rotation error by 25%–27% and translation error by up to 32% over the strongest baseline, while matching it in video fidelity and attaining the best overall photometric quality and 3D consistency of dynamic content.

Primary area:
generative models
Keywords:
Camera-Controlled Video Generation, Video Re-shooting, Flow Matching, Novel-View Video Synthesis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_vV7bOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents Manifold4D, a video reshooting method that injects noisy latents of point-cloud renders into the initial state of flow matching while keeping the source video as a visual condition. The design aims to reduce conflicts between geometric guidance and appearance guidance, which the authors call a "trust dilemma." Experiments on multiple benchmarks show better camera control with competitive visual quality.

Critical strengths
  1. Clear motivation and a simple design. The tension between source-video and point-cloud-render conditioning is well motivated. Injecting geometry through the initial state of flow matching, rather than as an extra condition, is a simple and natural way to address it.
  2. Meaningful gains in camera control. Manifold4D consistently improves camera-control accuracy. In the architecture ablation, it reduces rotation and translation errors by about 6.9% and 10.7% relative to the Vista4D-style variant.
  3. Extensive evaluation. The experiments cover camera control, visual quality, and dynamic-scene consistency across multiple benchmarks.
Critical weaknesses
  1. The contribution of the proposed mechanism is not cleanly isolated from the geometry estimator. All methods receive the same VGGT-Ω point-cloud projections at inference, but their training geometry differs. Manifold4D is trained mainly on VGGT-Ω, whereas Vista4D uses STream3R and π³. Shared inference inputs therefore do not imply equal adaptation to the distribution of VGGT-Ω artifacts. The supplementary π³ experiment changes the pose estimator used for evaluation, not the input geometry, so it does not resolve this confound. The architecture ablation (Sec. 4.4) shares the backbone, seed, and training budget, but compares only early-stage checkpoints. Its gains over the Vista4D-style variant are moderate and come with slightly lower visual-quality scores, so it remains unclear whether the advantage persists after full training.
  2. The changes to the training data are not justified or ablated. Compared with Vista4D, Manifold4D adds DL3DV, DynPose, and HuMMan, changes the dataset subsets, and changes the sampling proportions. Appendix B states that DL3DV and HuMMan have little influence on final quality, but no quantitative ablation supports this statement. The contribution of the data changes therefore cannot be separated from that of the geometry-injection mechanism.
  3. The central "trust dilemma" claim is not directly supported. The experiments do not show that conflicting visual conditions cause the observed failures, or that resolving this conflict explains the improvement. Another explanation is that initializing generation from a target-aligned render itself makes geometry easier to preserve.
Questions for authors
  1. Could you compare Manifold4D with Vista4D (or the Vista4D-style variant) under the full training budget, using identical training data and geometry preprocessing? Alternatively, could you evaluate both methods with Vista4D's original π³-based input geometry? If neither is feasible, please explain why the differences in training geometry do not affect the attribution of the gains.
  2. Why were DL3DV, DynPose, and HuMMan added, and why were the subsets and sampling proportions changed? Please provide a quantitative dataset ablation that separates the effect of the training data from that of the proposed mechanism.
  3. What targeted evidence shows that the "trust dilemma" causes the baseline failures, and that resolving it, rather than simply initializing from a target-aligned render, accounts for the improvement? For example, does a baseline that is also initialized from the render but keeps both conditions recover most of the gain?
4 comments
AuthorsOpenReview ↗

W1 / Q1: Dependence on the training geometry; budget of the architecture ablation

We thank the reviewer for these points; below we vary the input geometry as well and clarify the training budget of the architecture ablation.

Input geometry from other reconstructors. We agree that changing only the pose evaluator does not address this confounder; we therefore additionally vary the input geometry. The full study is given in our general response (“General Response to All Reviewers: Dependence on VGGT-Omega and consistency across pose estimators”): we condition every method on point clouds reconstructed with Pi3 and Depth-Anything-3 (DA3), and evaluate each output with VGGT-Omega, Pi3 and DA3, giving 3 × 3 settings in total (Table G1 there). In short, our rotation error is the lowest in all 9 settings, and with the well-posed target-aligned protocol our translation error is the lowest in all 9 settings. In particular, with Vista4D's own Pi3 geometry, our rotation error is 2.84° versus 3.44° for Vista4D under VGGT-Omega evaluation, and 2.83° versus 3.44° under Pi3 evaluation. Therefore, although only VGGT-Omega is used for training and evaluation, Manifold4D still generalizes well, and none of the mild self-consistency effects observed across evaluators changes the final ranking.

Below we show only the most important numbers; the full analysis and the complete 3 × 3 results, including the source-fitted scale, are given in our general response.

Table R1. Camera control with different input geometries and pose evaluators (Vista4D-Eval, 103 clips, seed 42). Each cell lists GEN3C / Vista4D / Ours.

    
Point cloudEvaluatorRotErr ↓TransErr ↓ (target-aligned)
GEN3C | Vista4D | OursGEN3C | Vista4D | Ours
VGGT-ΩVGGT-Ω3.97   3.74   2.650.041   0.043   0.031
Pi34.55   3.86   3.050.053   0.041   0.033
DA34.80   4.96   4.160.044   0.046   0.043
Pi3VGGT-Ω3.72   3.44   2.840.048   0.044   0.030
Pi33.89   3.44   2.830.053   0.045   0.032
DA34.29   4.80   4.200.051   0.050   0.040
DA3VGGT-Ω4.94   5.22   4.070.045   0.049   0.039
Pi34.60   5.09   4.320.054   0.047   0.039
DA35.34   5.96   5.100.051   0.050   0.047

Training budget of the architecture ablation. We ended the architecture ablation once the metrics had nearly plateaued. Beyond 15k steps, TransErr no longer improves and RotErr improves by only 0.2–0.4°. Among the variants, our design departs the most from the pretrained prior, since the render is injected into the starting noise rather than given as an additional condition. Even so, it has essentially converged by 15k steps, so we consider this budget sufficient for the comparison. Fully training one 14B variant takes about 2,400–3,600 A100 GPU-hours, so training every variant to the full schedule was beyond our budget.

AuthorsOpenReview ↗

W2 / Q2: Changes to the training data

Training on a different data mixture is common in this line of work: each baseline we compare with uses its own. ReCamMaster trains on its own synthetic multi-camera dataset, TrajectoryCrafter on OpenVid together with DL3DV and RealEstate10K, GEN3C on RealEstate10K, DL3DV, Waymo Open and Kubric4D, and Vista4D on OpenVid-HD together with MultiCamVideo.

Separating the data from the mechanism. The architecture ablation (Table 5(c)) is designed to address exactly this question. Its three variants use the same training data, sampling ratios, geometry preprocessing, training strategy and budget, and differ only in how the render enters the model. Our design still gives the lowest camera error, so the gain from the mechanism does not depend on the data. The control in Table R2 below also shares the same data.

Why these datasets. DL3DV contains static scenes with large camera motion and is also used by TrajectoryCrafter and GEN3C. We mainly included it because it was readily available for the early validation of our pipeline. DynPose provides monocular dynamic videos with substantial camera motion, while many OpenVid clips are captured with a nearly static camera. HuMMan provides real multi-view captures of moving humans, which we added to improve generalization to real videos, since MultiCamVideo is synthetic. Despite the additional sources, our training set is smaller than Vista4D's. Vista4D trains on a 60K-clip subset of OpenVid-HD together with MultiCamVideo, whereas we use 14.9K OpenVid-HD clips and 22.9K real monocular clips in total (OpenVid-HD and DynPose), about 36K clips over all sources.

The statement in Appendix B. It was based on qualitative observations rather than a controlled ablation. For DL3DV, we use the first half of each video as the source and the second half as the target, which we later found often places the two cameras so far apart that the render is very sparse. HuMMan is captured in a fixed studio, with little motion of the subject. We therefore expected both datasets to contribute little, but as the reviewer points out, we have no quantitative evidence for this. We will remove this sentence in the revision.

AuthorsOpenReview ↗

W3 / Q3: Direct evidence for the trust dilemma

We add two experiments. The first shows that the baselines indeed face the trust dilemma. The second shows that our gain does not come only from initializing generation from the target-aligned render, but also from resolving this dilemma.

Probing the trust of Vista4D in the render. The attention Vista4D pays to the render is highly uneven, and this unevenness predicts its failure. Recording the attention from the target tokens to the render tokens (averaged over all 40 layers), we find that its share on the dynamic region varies by 2.1× across the 110 Vista4D-Eval clips. Clips where the model attends less to the render show clearly worse camera control: the 20 clips with the lowest share have a mean RotErr of 5.02°, versus 3.06° for the remaining clips, and all of the 10 lowest show clear 3D inconsistency of the dynamic subject. Vista4D's trust in the render is therefore uncontrolled, and low trust directly hurts controllability.

Forcing more trust. If low attention to the render is the problem, forcing more attention should help, and it does, up to a point. On these 20 clips, we multiply the attention from the target tokens to the render tokens by a constant weight in every layer and step, and renormalize (Table R2). A larger weight steadily improves camera control (RotErr 6.15° → 4.30°), but it also steadily degrades visual quality (AQ 59.4 → 57.4). Even at the largest weight, the boosted Vista4D remains clearly behind ours on the same 20 clips (RotErr 4.30° versus 3.02°, and target-fitted TransErr 0.092 versus 0.080).

This remedy is, moreover, neither learned nor universal: it is a manual inference-time bias, and it does not hold across scenes (RotErr gets worse on 4–8 of the 20 clips). Too little trust leaves the dynamic subject inconsistent; forcing more trust degrades the video. No single level of trust works for every scene. This is the trust dilemma that a model conditioned on the render has to resolve, and our model avoids it by starting generation from the render instead of attending to it.

Table R2. Forcing Vista4D to attend more to the render (Vista4D-Eval, the 20 clips with the lowest render attention, VGGT-Omega point clouds, seed 42).

      
SettingRotErr ↓TransErr ↓ (target-aligned)TransErr ↓ (source-fitted)AQ ↑IQ ↑
Vista4D6.150.0410.26759.467.2
Attention weight ×1.55.990.0310.24859.066.8
Attention weight ×25.530.0270.24757.966.7
Attention weight ×44.300.0240.23957.465.7

Disentangling the initialization from the persistent render. We agree that starting from the target-aligned render by itself helps preserve geometry, and it is one of the two parts of our design. To measure how much of our gain comes from it, we train the control suggested by the reviewer: the GEN3C-style variant, which concatenates the render along the channel axis at every step, additionally initialized from our geometric noise. Compared with the GEN3C-style variant, it isolates the gain from the initialization; compared with ours, the gain from removing the render as a persistent input. All variants share the same source-video stream, training data and training budget.

Table R3. Disentangling the render-based initialization from the persistent render (Vista4D-Eval, 103 clips, 50 sampling steps, seed 10027; all variants trained for the same 15k steps).

        
VariantRender-based initializationPersistent render inputRotErr ↓TransErr ↓ (target-aligned)TransErr ↓ (source-fitted)AQ ↑IQ ↑
GEN3C-style (channel concat)✗✓4.960.0440.44854.769.0
Channel concat + render-based initialization✓✓3.900.0380.48052.868.2
Ours✓✗3.100.0330.39153.769.6

Initializing from the render clearly improves the GEN3C-style variant, but a gap to ours remains when the render is kept as a persistent input. The gain of our method therefore comes from both aspects: starting from the geometry, and removing the persistent render so that the network does not have to weigh it against the source video. The failure modes in the paper are consistent with this conflict: the GEN3C-style variant over-relies on the render and leaves dynamic objects fragmented, with 1.6× our rotation error, whereas Vista4D under-trusts the render on dynamic subjects, leading to 3D inconsistency (Figs. 4, 6 and 8; Table R2).

Our design therefore also does not follow the render blindly: when the dynamic mask is removed and the render of the moving subject is corrupted, it still recovers the correct motion from the source video (Fig. 7). We will add it to the architecture ablation.

Thanks again for the comments — our response is posted above, and we would be happy to address any remaining point you have. We look forward to hearing from you.

Reviewer_qgCQOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Manifold4D is a video reshooting (video novel-view synthesis) method that is conditioned on point cloud renders of the input/source video lifted into 4D and rendered at the target camera trajectory, along with the source video itself. Unlike existing methods which implicitly address conflicting visual cues from source video and point cloud render, which the paper calls the trust dilemma, Manifold4D addresses it by only conditioning the video flow matching model on the point cloud render at the first flow matching step by mixing the initial Gaussian noise with the point cloud render at non-hole (alpha non-transparent) regions, while conditioning on the source video in-context (like previous methods) at every flow matching step. Doing so allows the model to better follow the low-frequency structure of the point cloud render while ensuring that point cloud artifacts don't leak through into the final generated video. Quantitatively, Manifold4D achieves better camera control and matching video fidelity compared to baselines.

Critical strengths
  1. Good visual fidelity and camera control: Manifold4D showcases qualitative examples of more precise camera control than many of the baselines, all the while being able to address the trust dilemma problem while largely not being affected by 4D reconstruction point cloud artifacts. These are shown through extensive evaluations on multiple different datasets for camera control (also NVS and video fidelity). For example, the camera control and video fidelity quantitative results against different motion amplitudes (Figure 4) is especially well done and clearly showcases the strength of the model.
  2. Simple geometry conditioning idea: Manifold4D proposes a simple idea to condition on a geometry render of the target video from a 4D-lifted source video that addresses the trust dilemma problem. Geometry-bearing noise at the first timestep is an elegant way to ensure there are no conditioning/visual cue conflicts between the point cloud render and source video.
Critical weaknesses
  1. Confusing conditioning and residual noise ablations: Table 5 is framed as an inference time ablation but is in general not super meaningful as a proper ablation to see how varying the conditioning types and residual noise actually matters to training the model.
    • Camera embeddings: For example, "w/o camera emb." really just shows that the model never actually learned to use the injected camera embeddings and thus probably don't need to be trained with them to achieve the same quality. This is sensible because often times camera conditioning just requires low-frequency information that is fully contained within the geometry-bearing starting noise. This might have also been why GEN3C and TrajectoryCrafter both don't condition on camera embeddings. Though camera embeddings were included due to render dropouts during training as the authors mentioned, these inference-time ablations show that instead of actually learning to use the camera embeddings, the model just seem to largely ignore them nonetheless, at least with the current eval.
      • Do camera embeddings help in cases where the camera trajectory does include high-frequency motion, for example, a jittery hand-held camera with tons of shaking? Or are the camera embeddings basically completely ignored in Manifold4D because the video model doesn't need the embeddings to gain sufficient camera information and thus just ignores it? One potential way to test this is to just remove the camera embedding injection and see if the camera control is any different for jittery target trajectories where camera embeddings would provide useful information.
    • Other conditions: Removing text prompts, source videos, and injected point cloud renders doesn't actually tell us whether they are a required part of the model or isolate the effectiveness of each condition. A better ablation here would be training each of these with the same recipe as the final model and see if any of them are necessary.
    • Residual noise strength: The paper does not state what σ\sigma (residual noise strength) the model was trained with. 5(b) is a residual noise strength sweep but is at inference-time as Section 4.4 implies. (If 5(b) is not inference time, then this section and table is written very confusingly.) It seems more sensible to train the model with a fixed σ\sigma, so 5(b) is a bit confusing because the σ\sigma that would do best at inference time would likely just be the σ\sigma that the model is trained with. A proper ablation would be training the model at different fixed σ\sigma's and seeing which one results in the best camera control and video fidelity.
    • Architecture ablations are not inference-time: Table 5 makes 5(c), architecture ablation, seem like an inference-time ablation, which it is not. The architecture ablation should be its own table with its own captions instead of being grouped in with 5(a) and 5(b).
  2. Choice of pose estimator seems to lessen Manifold4D's camera control lead: Comparing Table 1 to Table 7, the camera control accuracy lead of Manifold4D when compared to baselines seems to become noticeably smaller when the pose estimator used for evaluation switches from VGGT-Omega to Pi3 (was this with Pi3 or Pi3X?). Manifold4D still performs generally the best, but Vista4D performs better on translation error on Vista4D-Eval, and GEN3C performs best on intrinsics error on DAVIS-Traj and on rotation error on raw Vista4D-Eval. On DAVIS-Traj rotation error, the point cloud render itself goes from tying Manifold4D (1.09 vs. 1.09) to beating it (1.37 vs. 1.56). VGGT-Omega is used both to build the point cloud render that Manifold4D's generation starts from and to estimate the poses in Table 1. The Table 1 numbers may therefore partially reward outputs that stay close to VGGT-Omega's reconstruction, which Manifold4D is especially prone to since it starts from it (and Manifold4D was also trained mostly with VGGT-Omega reconstructions, see Questions For Authors Q1 for more details). This matters because camera control is the main advantage Manifold4D claims over Vista4D, which it otherwise only matches in video fidelity. Evaluation results with pose estimators beyond Pi3 and VGGT-Omega would help clarify what exactly is causing this decrease in Manifold4D's lead.
Questions for authors
  1. Using the same training and inference 4D reconstructor: Manifold4D sees VGGT-Omega reconstructions in both training and inference. (The only non-VGGT-Omega reconstruction during training is HuMMan.) Thus, there may be at least a slight 'overfit' to VGGT-Omega's 4D reconstruction artifacts/idiosyncrasies that the current evaluation cannot reveal. This contrasts with the paper's two strongest baselines, GEN3C (where a variety of reconstruction methods + GT were used for preprocessing the data) and Vista4D (where training used STream3R as the main multiview reconstructor and inference used Pi3). Both mainly use a different 4D reconstructor at inference than at training, and in this paper's experiments both are also given VGGT-Omega renders, a reconstructor they were not trained on. That implicitly shows better generalization to different 4D reconstruction methods at inference. Can the authors show results where the input point cloud is built with a different 4D reconstruction method, such as Pi3/Pi3X or Depth Anything 3?
  2. Fitting camera trajectories from eval datasets: Vista4D-Eval provides reconstruction from Pi3 and hand-crafted camera trajectories. VGGT-Omega probably has a different scene scale than Pi3, so how were the camera trajectories of Vista4D-Eval fitted to the scene scale of VGGT-Omega? The same question also goes for how DAVIS-Traj had different scales for VGGT-Omega evals and Pi3 evals. This is important to ensure that Table 1 and 7 are actually evaluating examples of the same camera-scene alignment.
  3. What is the cost of starting from a non-Gaussian: The abstract claims that "the render is thus used exactly once, and the network is never asked to learn how to read it" (L43–44). However, starting the flow matching process from a non-Gaussian, when the base model was trained on a Gaussian prior, has consequences for the entire flow matching trajectory, not just the start. The paper's ablations don't isolate this particular way of injection. Table 5(c) does train a GEN3C-style variant, but it only contrasts geometry-bearing noise without a persistent render against (presumably) a Gaussian start with a persistent render. Missing controls include GEN3C- or TrajectoryCrafter-style channel concatenation applied only at the first or first few flow matching timesteps, and channel concatenation combined with the geometry-bearing starting noise. As mentioned in Weakness 1, the paper also doesn't seem to ablate training with different fixed σ\sigma. These are important because the paper's contribution is the claim that geometry-bearing noise at the flow matching start is the right place to inject pixel-aligned geometry (to address the trust dilemma). Currently, we don't know whether the gains vs. GEN3C and TrajectoryCrafter actually come from the in-context source video, or whether the gains vs. Vista4D actually come from only injecting the point cloud render near the starting timestep(s) (as MoCam also independently observed). Can the authors add these controls (concatenation only at the first few timesteps, concatenation + geometry-bearing noise, and training at different fixed σ)?
  4. Details on architecture ablation: Just to clarify what the authors meant by "train the conditioning layouts of the two strongest baselines ..." (L465), are these models trained in their exact conditioning types (point cloud render and source video in-context-concatenated for Vista4D and point cloud render fused at the video DiT start for GEN3C, both with pure Gaussian noise as the starting target output tokens), or are we only keeping their architecture and training with geometry-bearing starting noise for their output tokens?
  5. Comparisons to NeoVerse: This is not super critical, but NeoVerse (CVPR 2026) [1] is concurrent work to Vista4D and also directly addresses the trust dilemma problem by perturbing the geometry rendering (4DGS) during training so the video model is robust to 4DGS artifacts. (The framing of training the model to be robust to geometry artifacts by training with artifacts is similar to that of Vista4D.) If the authors have bandwidth, it would be good to see comparisons of Manifold4D to NeoVerse as well.

[1] Yuxue Yang, Lue Fan, Ziqi Shi, Junran Peng, Feng Wang, Zhaoxiang Zhang. "NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos." CVPR 2026.

6 comments
AuthorsOpenReview ↗

W2 / Q1 / Q2: Dependence on the pose estimator and the 4D reconstructor; scale of the target trajectories

We thank the reviewer for these questions, which prompted us to re-examine how the target trajectories are compared across estimators and why the translation ranking changes.

Camera frame and scale of the target trajectories (Q2). Feed-forward reconstructors such as VGGT-Omega and Pi3 (we use the original Pi3, not Pi3X) recover each scene only up to its own scale and coordinate frame. Even one model already behaves this way: VGGT-Omega gives different frames and scales for the source video alone and for the source-plus-generated reconstruction, so the estimated trajectory of the generated video cannot be compared with the target directly.

Concretely, the target trajectory is defined in the frame and scale of a standalone reconstruction of the source video. After generation, we jointly reconstruct the source and generated videos, which gives a new frame and scale. Both reconstructions contain the source cameras, so we estimate a Sim(3) transform between them using the source-video cameras alone, in closed form: the rotation R is obtained by aligning the source-camera orientations, the translation t by matching the mean of the source-camera centres, and the scale s by the ratio of the spreads of the source-camera centres in the two reconstructions, giving p → sRp + t. Applying the same (s, R, t) to the estimated generated trajectory removes the difference in frame and scale, so the aligned trajectory can be compared with the target trajectory directly (Appendix D).

Why the translation ranking differs across estimators. This protocol estimates the scale from the source camera, which barely moves in 77% of Vista4D-Eval clips (less than 5% of the scene depth), so the scale is ill-posed: even with VGGT-Omega on both sides it is off by more than 2× in about one fifth of the clips. Aligning instead the generated trajectory directly to the requested target trajectory, as in the standard Sim(3) absolute-trajectory-error protocol, is well conditioned on every clip, because the requested target trajectories always contain substantial camera motion. With this target-aligned protocol, our translation error is the lowest in all 9 settings (Table G1 of the general response).

Input geometry and pose estimator (Q1, W2). The full study is given in our general response (“General Response to All Reviewers: Dependence on VGGT-Omega and consistency across pose estimators”): we condition every method on point clouds reconstructed with Pi3 and Depth-Anything-3 (DA3) and evaluate each output with VGGT-Omega, Pi3 and DA3. Our rotation error is the lowest in all 9 settings, including the 4 that never use VGGT-Omega, and with the target-aligned protocol our translation error is the lowest in all 9. Although only VGGT-Omega is used in training, Manifold4D generalizes well, and no self-consistency effect across evaluators changes the final ranking.

Additional analyses. We further explain why the rotation-error advantage of Manifold4D over Vista4D shrinks under the VGGT-Omega and Pi3 evaluators in the paper: one failure case left in the 106-clip set inflates the mean of both methods, and of ours the most. Table G2 of the general response reports the median rotation error over all 110 clips, which is insensitive to such extreme values and needs no manually chosen outlier threshold: ours is the lowest in 8 of the 9 settings and better than Vista4D in all 9, and the margin stays large under the VGGT-Omega and Pi3 evaluators and narrows under DA3, which we attribute to the weaker robustness of DA3's geometry estimation on generated videos. Table G3 reports the same 3 × 3 study on DAVIS-Traj, whose source cameras move substantially (lowest RotErr in all 9 and lowest TransErr in 8).

Below we show only the most important numbers;the full analysis and the complete 3 × 3 results are given in our general response.

Table R1. Camera control with different input geometries and pose evaluators (Vista4D-Eval, 103 clips, seed 42). Each cell lists GEN3C / Vista4D / Ours.

    
Point cloudEvaluatorRotErr ↓TransErr ↓ (target-aligned)
GEN3C | Vista4D | OursGEN3C | Vista4D | Ours
VGGT-ΩVGGT-Ω3.97 3.74 2.650.041   0.043   0.031
Pi34.55 3.86 3.050.053   0.041   0.033
DA34.80 4.96 4.160.044   0.046   0.043
Pi3VGGT-Ω3.72 3.44 2.840.048   0.044   0.030
Pi33.89 3.44 2.830.053   0.045   0.032
DA34.29 4.80 4.200.051   0.050   0.040
DA3VGGT-Ω4.94 5.22 4.070.045   0.049   0.039
Pi34.60 5.09 4.320.054   0.047   0.039
DA35.34 5.96 5.100.051   0.050   0.047

We will add these analyses to the appendix, clarify the alignment of the target trajectories, and revise the Pi3 paragraph accordingly.

AuthorsOpenReview ↗

Q3: Cost of starting from a non-Gaussian distribution; missing controls

Cost of the non-Gaussian start. We agree that injecting the render into the starting noise breaks the Gaussian prior of the base model, and that this affects the whole flow-matching trajectory rather than only its first step. In practice, the cost we observe lies in optimization: our model performs poorly early in training and converges more slowly than the variants that keep a Gaussian start, since it has to adapt to a shifted starting distribution. Once trained, however, it attains better camera control than these variants with comparable visual quality (Table 5(c)). By "the network does not need to learn to read it" (L43–44), we mean that the render is not an extra input that the network must learn to interpret, as it is the starting point of generation.

Additional control. We thank the reviewer for these valuable suggestions. With limited compute (fully training one 14B variant takes about 2,400–3,600 A100 GPU-hours), we cannot train every proposed variant, so we chose the most informative one: concatenation plus geometric noise. It takes the GEN3C-style variant, where the render is concatenated along the channel axis at every step, and additionally starts generation from our geometric noise. Compared with the GEN3C-style variant, it measures how much of our gain comes from starting from the geometric noise. Compared with ours, it measures how much comes from removing the render as a persistent input, i.e., from resolving the trust dilemma. We build on the GEN3C-style variant rather than the Vista4D-style one because the latter takes about 1.5× longer to train. All variants share the same in-context source-video stream, training data and training budget, so the differences do not come from the source video.

Table R2. Disentangling the geometric-noise start from the persistent render (Vista4D-Eval, 103 clips, 50 sampling steps, seed 10027).

        
VariantGeometric-noise startPersistent render inputRotErr ↓TransErr ↓ (target-aligned)TransErr ↓ (source-fitted)AQ ↑IQ ↑
GEN3C-style (channel concat)✗✓4.960.0440.44854.769.0
Channel concat + geometric noise✓✓3.900.0380.48052.868.2
Ours✓✗3.100.0330.39153.769.6

As shown in Table R2, with the same 15k training budget the geometric-noise start clearly improves the GEN3C-style variant (RotErr 4.96° → 3.90°), but it still falls behind ours (3.10°), with lower visual quality as well. The gain of our method therefore comes from both aspects: starting generation from the geometry, and removing the render as a persistent input so that the network is not forced to keep trusting it. We will add it to the architecture ablation.

AuthorsOpenReview ↗

W1/Q4/Q5: Ablation details, comparison with NeoVerse, and other notes on the ablation tables

Q4: Details of the architecture ablation. Both variants keep the conditioning type of the original baseline and start from pure Gaussian noise rather than from our geometric noise. Specifically, the Vista4D-style variant uses a three-stream token layout, with the render and the source video given as separate in-context streams next to the output tokens. The GEN3C-style variant concatenates the render to the output tokens along the channel axis and keeps the source video in context.

Q5: Comparison with NeoVerse. We thank the reviewer for the pointer. NeoVerse is closely related to our work. We ran the official NeoVerse release on Vista4D-Eval, with its own 4D reconstruction at its native resolution (Table R3). Like Vista4D, it makes the video model robust to geometric artifacts by training on degraded 4DGS renders, but it appears to be designed mainly for small viewpoint changes. On the third of the clips with the smallest viewpoint change, its rotation error is 3.32°, but on the third with the largest change it rises to 15.68°, and the content it generates in disoccluded regions is of lower quality. We will discuss NeoVerse in the related work.

Table R3. Comparison with NeoVerse (Vista4D-Eval, seed 42; camera metrics on 103 clips, quality on 110 clips).

       
MethodRotErr ↓RotErr ↓ (small / large view change)TransErr ↓ (target-aligned)TransErr ↓ (source-fitted)AQ ↑IQ ↑
NeoVerse8.463.32 / 15.680.0580.46352.968.3
Ours2.652.38 / 3.140.0310.37553.870.5

W1: Other notes on the ablation tables.

Camera embedding. We tested the setting with rapid camera motion, and the camera embedding still brings no improvement. Our interpretation is that once the render already provides very strong geometry, the model has little incentive to learn to use the camera injection, so the embedding remains inert.

Other conditioning. We agree that ablating the conditions only at inference time does not establish their necessity. This is a compute constraint: a training-time ablation for each conditioning choice is beyond our budget, so we ablate at inference to observe what each condition contributes to trajectory control and video quality.

On Table 5(c). This table is indeed a training-time ablation, and we agree that placing it next to the inference-time tables invites confusion; we will present it as a separate table.

On σ. σ is indeed fixed during training: an ODE needs a fixed starting point. We initially trained with σ = 0, which led to a copy-the-render degradation in the generated videos. Guided by this observation and by visual inspection, we chose σ = 0.3, which visibly preserves the geometric structure while leaving room for the model to modify it. Below we report our earlier σ = 0 vs σ = 0.3 comparison. The σ sweep in the paper serves two purposes: to demonstrate robustness, and to calibrate a hyperparameter over its inference range. A training-time sweep to maximize over σ is beyond our compute budget.

Table R4. Training-time σ control (DAVIS-Traj, 72 clips, seed 42).

      
Training σRotErr ↓TransErr ↓ (target-aligned)TransErr ↓ (source-fitted)AQ ↑IQ ↑
0.3 (used)1.150.0350.11251.165.5
01.140.0340.19251.164.7
Reviewer_YoMdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Manifold4D, a framework for video re-shooting with better camera trajectory control. The key idea is to inject the point cloud rendered video into the initial noise of flow matching, instead of using it as an additional condition during the whole denoising process. Specifically, the authors first reconstruct a 4D point cloud from the source video and render it with the target camera trajectory. The rendered video is then combined with Gaussian noise as the initial latent, while the source video provides visual condition during denoising. The authors further fine-tune Wan2.1-T2V with the modified flow matching objective. Experiments show that the proposed method achieves better camera control accuracy while maintaining comparable visual quality.

Critical strengths
  1. The paper addresses an interesting and practical problem in camera-controlled video generation. The motivation of avoiding the conflict between point cloud rendering and source video is reasonable.

  2. The proposed geometry-bearing noise initialization is a simple but effective design. It achieves better camera control without introducing additional rendering condition during denoising.

  3. The experiments are relatively comprehensive, including comparisons with existing methods, architecture ablations, and evaluation under larger camera motions. The improvement on camera trajectory control is impressive.

Critical weaknesses
  1. I have some concerns about the camera control evaluation. The authors use VGGT-Omega for both point cloud reconstruction and camera pose evaluation, which may introduce some bias. Although the authors also provide evaluation with Pi3, the results are not fully consistent. For example, Vista4D achieves better translation error than the proposed method on Vista4D-Eval under Pi3 evaluation (0.382 vs. 0.419). This makes me question how reliable the reported improvements are across different pose estimation methods.

  2. Table 5(c) provides comparisons with Vista4D-style and GEN3C-style architectures, which I think is a useful ablation. However, these variants are only trained with the same early-stage training budget instead of the full training schedule. It is unclear whether the improvement can still hold with sufficient training. Besides, the robustness to inaccurate point cloud geometry is mainly demonstrated with qualitative results. I believe a more comprehensive quantitative evaluation would be helpful.

Questions for authors
  1. Could the authors further explain the potential bias caused by using VGGT-Omega for both reconstruction and evaluation? Especially considering that Pi3 produces different results on some metrics.

  2. How sensitive is the proposed method to the quality of the reconstructed point cloud? For example, when there are large depth errors, inaccurate dynamic masks, or missing geometry, how would these errors affect the final generation quality?

3 comments
AuthorsOpenReview ↗

W1 / Q1: Bias from using VGGT-Omega for both reconstruction and evaluation; different translation results under Pi3

We thank the reviewer for this insightful question. Prompted by it, we investigated why the translation error, unlike the rotation error, varies across pose estimators. The full study is given in our general response (“General Response to All Reviewers: Dependence on VGGT-Omega and consistency across pose estimators”); below we summarize its conclusions.

Summary. The translation difference across evaluators comes from the ill-posed scale estimation of the evaluation protocol: the scale is estimated by aligning the source video, which is nearly static in most clips. With the well-posed target-aligned protocol (the generated trajectory is aligned directly to the requested trajectory, as in the standard Sim(3) ATE protocol), our translation error is the lowest in all 9 settings. We further vary the input geometry (Pi3, DA3) together with the pose evaluator (VGGT-Omega, Pi3, DA3), giving 3 × 3 settings in which both our rotation error and our translation error are the lowest in all 9, which indicates that our advantage is stable across pose estimators and does not depend on the one used in the paper. Details and per-setting numbers are given in our general response (Tables G1–G3).

Additional analyses. We further explain why, in the paper, the rotation-error advantage of Manifold4D over Vista4D shrinks under the VGGT-Omega and Pi3 evaluators: one failure case that the 106-clip set fails to remove inflates the mean of both methods, and of ours the most. Table G2 of the general response reports the median rotation error over all 110 clips. A median is insensitive to the extreme values produced by mis-aligned clips, and it also removes the unfairness of a manually chosen outlier threshold; with it ours is the lowest in 8 of the 9 settings and better than Vista4D in all 9. The margin stays large under the VGGT-Omega and Pi3 evaluators and narrows under DA3, which we attribute to the weaker robustness of DA3's geometry estimation on generated videos rather than to a smaller advantage of our method. Table G3 reports the same 3 × 3 study on DAVIS-Traj, whose source cameras move substantially (lowest RotErr in all 9 and lowest TransErr in 8).

Below we show only the most important numbers; the full analysis and the complete 3 × 3 results are given in our general response.

Table R1. Camera control with different input geometries and pose evaluators (Vista4D-Eval, 103 clips, seed 42). Each cell lists GEN3C / Vista4D / Ours.

    
Point cloudEvaluatorRotErr ↓TransErr ↓ (target-aligned)
GEN3C | Vista4D | OursGEN3C | Vista4D | Ours
VGGT-ΩVGGT-Ω3.97 3.74 2.650.041   0.043   0.031
Pi34.55 3.86 3.050.053   0.041   0.033
DA34.80 4.96 4.160.044   0.046   0.043
Pi3VGGT-Ω3.72 3.44 2.840.048   0.044   0.030
Pi33.89 3.44 2.830.053   0.045   0.032
DA34.29 4.80 4.200.051   0.050   0.040
DA3VGGT-Ω4.94 5.22 4.070.045   0.049   0.039
Pi34.60 5.09 4.320.054   0.047   0.039
DA35.34 5.96 5.100.051   0.050   0.047

We will add this analysis to the appendix and revise the Pi3 paragraph accordingly.

AuthorsOpenReview ↗

W2 / Q2: Training budget of the variants and sensitivity to point cloud quality

Training budget.

We ended the architecture ablation once the metrics had nearly plateaued. Beyond 15k steps, TransErr no longer improves and RotErr improves by only 0.2–0.4°. Among the variants, our design departs the most from the pretrained prior, since the render is injected into the starting noise rather than given as an additional condition. Even so, it has essentially converged by 15k steps, so we consider this budget sufficient for the comparison. Fully training one 14B variant takes about 2,400–3,600 A100 GPU-hours, so training every variant to the full schedule was beyond our budget.

Sensitivity to point cloud quality.

We add a quantitative experiment on 40 randomly selected Vista4D-Eval clips, in which we manually degrade the point cloud given to our model in three ways (Table R1): (i) the most extreme mask error, where we remove the dynamic mask entirely, so all dynamic points are treated as static and accumulated across all frames; (ii) missing geometry, where 10% of each image (blob-shaped regions) has no geometry; and (iii) depth error, where the depth of each frame is scaled by a smoothly varying factor with about 3% and 5% error. Vista4D and GEN3C are given the clean point cloud. Without the mask or with missing geometry, camera accuracy and visual quality degrade only mildly, and our RotErr remains lower than that of Vista4D and GEN3C under all three evaluators. With 3% depth noise the degradation is similarly mild, and even with 5% depth noise manually introduced, our camera accuracy remains comparable to that of Vista4D and GEN3C with the clean point cloud. Our method therefore adapts well to lower-quality point clouds.

Table R1. Robustness to point cloud errors (40 randomly selected Vista4D-Eval clips, seed 42). Each camera cell: RotErr / TransErr (target-aligned). Vista4D and GEN3C use the clean VGGT-Ω point cloud.

      
SettingVGGT-Ω evalPi3 evalDA3 evalAQ ↑IQ ↑
Ours (clean point cloud)2.32 / 0.0242.61 / 0.0293.50 / 0.03553.670.2
Ours, 10% missing geometry2.95 / 0.0243.28 / 0.0254.10 / 0.03253.369.6
Ours, no dynamic mask3.25 / 0.0243.35 / 0.0314.28 / 0.03853.369.5
Ours, 3% depth noise3.45 / 0.0283.88 / 0.0374.84 / 0.04453.570.1
Ours, 5% depth noise4.19 / 0.0354.38 / 0.0335.49 / 0.04552.669.6
Vista4D (clean point cloud)3.70 / 0.0373.74 / 0.0394.61 / 0.04054.171.4
GEN3C (clean point cloud)4.11 / 0.0343.99 / 0.0364.59 / 0.03647.961.2

Effect on the dynamic subject. Visually, removing the mask does cause slight misalignment of the human body, where points from different frames overlap in the render. The trajectory error, however, rises only slightly. This is because pose estimators rely mainly on the static background, so trajectory error mostly reflects how closely the static regions follow the render (Sec. 4.3): simply copying the render, or following it closely on static regions as Vista4D does, already yields a low trajectory error. For this reason, beyond camera control, our paper makes a second central claim about the dynamic subject. Vista4D shows clear 3D inconsistency on dynamic humans, while methods that over-trust the sparse render, such as GEN3C, leave dynamic objects fragmented. Our design addresses both, as shown by the qualitative comparisons (Figs. 4, 6 and 8) and the user study (Table 4). In the user study the render is overlaid on each video to judge dynamic consistency, and ours is preferred in 62.4% of cases, versus 8.7% for Vista4D and 9.5% for GEN3C.

Thanks again for the comments — our response is posted above, and we would be happy to address any remaining point you have. We look forward to hearing from you.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers: Dependence on VGGT-Omega and consistency across pose estimators

All reviewers asked, in different forms, whether our gain in camera control depends on VGGT-Omega, which we use both to build the training geometry and to evaluate the generated videos (YoMd W1/Q1, qgCQ W2/Q1, vV7b W1/Q1); YoMd and qgCQ also asked why the translation error behaves differently under Pi3. Since the question is shared, we answer it here in full and summarize it in the individual responses. In short:

  • Rotation. Across 3 input reconstructors × 3 pose evaluators, our rotation error is the lowest in all 9 settings (Table G1).
  • Translation. The inconsistency across evaluators comes from the ill-posed alignment of the evaluation protocol, not from camera control: under the standard target-aligned protocol our translation error is the lowest in all 9 settings (Table G1).

Different 4D reconstructors and pose estimators. To test whether our gain comes from overfitting to VGGT-Omega, we additionally reconstruct the input point clouds with Pi3 (the geometry Vista4D uses) and Depth-Anything-3 (DA3), trained on by none of the methods, and condition every method on them. We then evaluate each output with VGGT-Omega, Pi3 and DA3, giving 9 settings (Table G1). Our rotation error is the lowest in all 9, and our target-aligned translation error in all 9 as well: although only VGGT-Omega is used for training and evaluation, Manifold4D generalizes well. A mild self-consistency effect exists—every method's rotation error drops when Pi3 is used for both reconstruction and evaluation (ours 3.05 → 2.83, Vista4D 3.86 → 3.44), and each method benefits from its own pretraining geometry (Manifold4D from VGGT-Omega, Vista4D from Pi3)—but none of these effects changes the final ranking.

Scale alignment. Pose estimators recover trajectories only up to an unknown scale, so the estimated generated trajectory must be aligned in scale before translation error can be computed. Following Vista4D, the protocol used in the paper (Appendix D) estimates this alignment from the source video: it aligns the source trajectory in the joint (source + generated) reconstruction to that of a standalone reconstruction of the source video, and applies the same transform to the generated trajectory (the source-fitted scale). In 77% of Vista4D-Eval clips, however, the source camera barely moves (less than 5% of the scene depth), so the scale is estimated from two nearly static trajectories and is ill-posed: even when both reconstructions use VGGT-Omega, it is off by more than 2× in about one fifth of the clips. This is likely what makes the methods unstable under the source-fitted evaluation.

The comparisons in this response therefore use a fully target-aligned protocol: the entire similarity transform is estimated by aligning the generated trajectory directly to the requested target trajectory, as in the standard Sim(3) ATE protocol (Sturm et al., TRO 2012). The requested trajectories always contain substantial camera motion, so this alignment is well conditioned. Rotation error is scale-invariant and is reported under the paper protocol. With this alignment the results become consistent across evaluators, with ours lowest in all 9 settings (Table G1). We will state this limitation in Appendix D and report both protocols.

Table G1. Camera control accuracy: 3 point cloud sources × 3 pose evaluators (Vista4D-Eval, 103 clips, seed 42). Cells list GEN3C / Vista4D / Ours; target-aligned TransErr is the primary metric.

     
Point cloudEvaluatorRotErr ↓TransErr ↓ (target-aligned)TransErr ↓ (source-fitted scale)
GEN3C | Vista4D | OursGEN3C | Vista4D | OursGEN3C | Vista4D | Ours
VGGT-ΩVGGT-Ω3.97   3.74   2.650.041   0.043   0.0310.433   0.410   0.375
Pi34.55   3.86   3.050.053   0.041   0.0330.442   0.358   0.373
DA34.80   4.96   4.160.044   0.046   0.0430.436   0.406   0.450
Pi3VGGT-Ω3.72   3.44   2.840.048   0.044   0.0300.453   0.369   0.385
Pi33.89   3.44   2.830.053   0.045   0.0320.439   0.373   0.368
DA34.29   4.80   4.200.051   0.050   0.0400.382   0.415   0.426
DA3VGGT-Ω4.94   5.22   4.070.045   0.049   0.0390.530   0.390   0.432
Pi34.60   5.09   4.320.054   0.047   0.0390.497   0.393   0.390
DA35.34   5.96   5.100.051   0.050   0.0470.447   0.429   0.446
AuthorsOpenReview ↗

General Response to All Reviewers: Dependence on VGGT-Omega and consistency across pose estimators

On the narrowing lead. Under VGGT-Ω evaluation (main paper) our rotation lead over Vista4D is 1.09° (2.65° vs 3.74°), whereas in the Pi3 appendix table it narrows to 0.25°: 3.82° vs 4.07° over the 106-clip common set, with GEN3C at 4.05°. Table G1 shows a much weaker narrowing: on the same Pi3 evaluator, over the 103-clip common set, ours is 3.05° vs 3.86° of Vista4D, a lead of 0.81°. The difference comes from one failure case that the 106-clip set fails to remove. It inflates the mean of both our method and Vista4D, and of ours the most. Tuning the threshold again is not a sound remedy, so to remove any dependence on a manually chosen threshold, Table G2 reports the median RotErr over all 110 clips without any filtering: ours is the lowest in 8 of the 9 settings and better than Vista4D in all 9, with a stable margin of 0.3–1.2° over the 9 settings (median 0.8°). The margin stays large under the VGGT-Ω and Pi3 evaluators and narrows under DA3 (0.33–0.34° in the two settings where DA3 evaluates the VGGT-Ω and Pi3 point clouds), which we attribute to the weaker robustness of DA3's geometry estimation on generated videos rather than to a smaller advantage of our method. The narrowing in the appendix is thus caused by extreme values that were not removed, not by a genuine loss of accuracy.

Table G2. Median RotErr over all 110 clips without filtering (Vista4D-Eval, seed 42). Each cell lists GEN3C / Vista4D / Ours.

   
Point cloudEvaluatorRotErr ↓ (median)
GEN3C | Vista4D | Ours
VGGT-ΩVGGT-Ω2.61   3.07   2.15
Pi33.17   3.23   2.25
DA33.72   3.92   3.58
Pi3VGGT-Ω2.68   2.70   2.00
Pi32.61   2.80   2.07
DA33.28   3.86   3.53
DA3VGGT-Ω3.63   4.01   2.90
Pi33.69   4.32   3.10
DA34.14   4.72   3.88

DAVIS-Traj. We also repeated the 3 × 3 study on DAVIS-Traj, whose source cameras move substantially. Ours attains the lowest RotErr in all 9 settings and the lowest TransErr in 8, and is better than Vista4D in all 9 settings on both metrics. Since the moving source cameras constrain the scale well there, the target-aligned and source-fitted protocols give consistent rankings, and we report both (Table G3), which is consistent with the analysis above.

Table G3. Camera control accuracy on DAVIS-Traj across 3 point cloud sources × 3 pose evaluators (72 clips, seed 42). Each cell lists GEN3C / Vista4D / Ours.

     
Point cloudEvaluatorRotErr ↓TransErr ↓ (target-aligned)TransErr ↓ (source-fitted scale)
GEN3C | Vista4D | OursGEN3C | Vista4D | OursGEN3C | Vista4D | Ours
VGGT-ΩVGGT-Ω1.30   2.53   1.080.048   0.083   0.0430.212   0.259   0.177
Pi31.91   2.83   1.560.055   0.082   0.0460.286   0.283   0.245
DA32.31   3.41   2.170.065   0.098   0.0690.295   0.326   0.305
Pi3VGGT-Ω1.40   2.40   1.230.053   0.092   0.0370.280   0.261   0.121
Pi31.60   2.53   1.520.056   0.080   0.0470.264   0.280   0.264
DA32.16   3.26   2.070.079   0.107   0.0680.348   0.335   0.319
DA3VGGT-Ω2.38   3.22   1.650.062   0.101   0.0450.352   0.276   0.167
Pi32.35   3.50   1.940.070   0.097   0.0470.319   0.340   0.264
DA32.75   3.82   2.500.071   0.102   0.0670.334   0.362   0.315

Beyond trajectory error. Our improvement over Vista4D is not limited to lower trajectory error: our method also achieves much stronger 3D consistency of dynamic objects. As shown in Figs. 4 and 6 and the supplementary videos, the dynamic objects generated by Vista4D often fail to follow the point cloud. The user study (Table 4) further confirms this: when the render is overlaid on each video to judge dynamic consistency, ours is preferred in 62.4% of cases, versus 8.7% for Vista4D.

Planned revisions. We will add Tables G1–G3 and this analysis to the appendix, state the limitation of the source-fitted alignment explicitly in Appendix D and report both alignment protocols, and revise the Pi3 paragraph accordingly.

Dear Reviewers,

As the discussion phase comes to an end, we'd appreciate it if you could take a moment to review the authors' responses. If you have any remaining concerns or questions, please raise them with the authors before the discussion closes.

Best,

Your AC

AuthorsOpenReview ↗

Summary of the discussion.

We thank the AC for coordinating the discussion. This note summarizes the main concern shared by the reviewers and our response.

1. Our response.

1.1 VGGT-Ω used for both reconstruction and evaluation. All three reviewers noted that our model builds its geometry with VGGT-Ω and is also evaluated with it, which could bias the comparison. We therefore rebuild the input point clouds with VGGT-Ω, Pi3 and Depth-Anything-3, and score every output with all three estimators, giving nine settings in which our method attains both the lowest rotation error and the lowest target-aligned translation error. The conclusion therefore does not depend on this choice.

1.2 Trust dilemma vs. geometric initialization. Reviewers vV7b and qgCQ asked whether our gain comes from injecting geometry into the starting point (A) rather than from resolving the trust dilemma (B). We added the ablation that has A but not B: the GEN3C-style variant is also initialized from the geometric noise. This improves its trajectory error, but it remains clearly above ours, so the gain comes from A and B together.

Reviewer vV7b also questioned whether the baselines actually face a trust dilemma; beyond the variant ablation above, we ran two additional experiments on Vista4D to test this directly. First, we analyzed the attention distribution of Vista4D and found that its attention to geometric information differs by 2.1× across the 110 clips; the 20 clips with the lowest geometric share have a mean rotation error of 5.02° versus 3.06° for the remaining clips, so its trust in geometry is not controlled. Second, scaling up this attention improves geometric control monotonically but degrades perceptual quality (rotation error 6.15° → 4.30°, AQ 59.4 → 57.4), and even at the largest weight it remains behind ours (4.30° vs. 3.02°). Together these results indicate that Vista4D does face the dilemma: its reliance on geometry is neither controlled nor recoverable by a single global weight, whereas our method resolves it by design.

1.3 On the dataset difference. Reviewer vV7b noted that our dataset differs from that of Vista4D and asked for an ablation showing that the gains do not stem from the dataset change. We clarify two points. First, every baseline is trained and evaluated on its own dataset, and dataset ablations are not standard practice in this line of work. Second, all of our ablation variants are trained on the same data with the same training recipe and differ only in architecture, so the gaps between them isolate the architectural contribution. Finally, the dataset ablation alone would exceed our compute budget. We also explain why our training data departs from Vista4D's.

1.4 Other points. Reviewer YoMd raised a concern about the sensitivity of our method to point-cloud quality; we address it with four quantitative experiments on perturbed and pruned point clouds, which show that our method is robust. Reviewer qgCQ asked for several clarifications and offered suggestions; we have answered all of them and revised the paper accordingly.

In sum, we have addressed every point raised and supported each response with additional experiments. Manifold4D is simple yet effective: it resolves the trust dilemma that earlier models face, attains the lowest trajectory error among the methods we compare, and we believe it makes a useful contribution to this line of work.

2. Outcome.

Reviewer YoMd considered his main concerns addressed and kept his rating. Reviewer qgCQ found that the two additional experiments addressed his concerns and raised his rating from 3 to 4. Reviewer vV7b has not provided a further response; his rating stands unchanged.

We are grateful to the AC for a careful handling of this discussion, and to the reviewers for their detailed reading and constructive criticism, which prompted the estimator sweep and the two Vista4D analyses reported above. All revisions described above have been incorporated into the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.