- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper presents Manifold4D, a video reshooting method that injects noisy latents of point-cloud renders into the initial state of flow matching while keeping the source video as a visual condition. The design aims to reduce conflicts between geometric guidance and appearance guidance, which the authors call a "trust dilemma." Experiments on multiple benchmarks show better camera control with competitive visual quality.
Critical strengths
- Clear motivation and a simple design. The tension between source-video and point-cloud-render conditioning is well motivated. Injecting geometry through the initial state of flow matching, rather than as an extra condition, is a simple and natural way to address it.
- Meaningful gains in camera control. Manifold4D consistently improves camera-control accuracy. In the architecture ablation, it reduces rotation and translation errors by about 6.9% and 10.7% relative to the Vista4D-style variant.
- Extensive evaluation. The experiments cover camera control, visual quality, and dynamic-scene consistency across multiple benchmarks.
Critical weaknesses
- The contribution of the proposed mechanism is not cleanly isolated from the geometry estimator. All methods receive the same VGGT-Ω point-cloud projections at inference, but their training geometry differs. Manifold4D is trained mainly on VGGT-Ω, whereas Vista4D uses STream3R and π³. Shared inference inputs therefore do not imply equal adaptation to the distribution of VGGT-Ω artifacts. The supplementary π³ experiment changes the pose estimator used for evaluation, not the input geometry, so it does not resolve this confound. The architecture ablation (Sec. 4.4) shares the backbone, seed, and training budget, but compares only early-stage checkpoints. Its gains over the Vista4D-style variant are moderate and come with slightly lower visual-quality scores, so it remains unclear whether the advantage persists after full training.
- The changes to the training data are not justified or ablated. Compared with Vista4D, Manifold4D adds DL3DV, DynPose, and HuMMan, changes the dataset subsets, and changes the sampling proportions. Appendix B states that DL3DV and HuMMan have little influence on final quality, but no quantitative ablation supports this statement. The contribution of the data changes therefore cannot be separated from that of the geometry-injection mechanism.
- The central "trust dilemma" claim is not directly supported. The experiments do not show that conflicting visual conditions cause the observed failures, or that resolving this conflict explains the improvement. Another explanation is that initializing generation from a target-aligned render itself makes geometry easier to preserve.
Questions for authors
- Could you compare Manifold4D with Vista4D (or the Vista4D-style variant) under the full training budget, using identical training data and geometry preprocessing? Alternatively, could you evaluate both methods with Vista4D's original π³-based input geometry? If neither is feasible, please explain why the differences in training geometry do not affect the attribution of the gains.
- Why were DL3DV, DynPose, and HuMMan added, and why were the subsets and sampling proportions changed? Please provide a quantitative dataset ablation that separates the effect of the training data from that of the proposed mechanism.
- What targeted evidence shows that the "trust dilemma" causes the baseline failures, and that resolving it, rather than simply initializing from a target-aligned render, accounts for the improvement? For example, does a baseline that is also initialized from the render but keeps both conditions recover most of the gain?
4 comments
W2 / Q2: Changes to the training data
Training on a different data mixture is common in this line of work: each baseline we compare with uses its own. ReCamMaster trains on its own synthetic multi-camera dataset, TrajectoryCrafter on OpenVid together with DL3DV and RealEstate10K, GEN3C on RealEstate10K, DL3DV, Waymo Open and Kubric4D, and Vista4D on OpenVid-HD together with MultiCamVideo.
Separating the data from the mechanism. The architecture ablation (Table 5(c)) is designed to address exactly this question. Its three variants use the same training data, sampling ratios, geometry preprocessing, training strategy and budget, and differ only in how the render enters the model. Our design still gives the lowest camera error, so the gain from the mechanism does not depend on the data. The control in Table R2 below also shares the same data.
Why these datasets. DL3DV contains static scenes with large camera motion and is also used by TrajectoryCrafter and GEN3C. We mainly included it because it was readily available for the early validation of our pipeline. DynPose provides monocular dynamic videos with substantial camera motion, while many OpenVid clips are captured with a nearly static camera. HuMMan provides real multi-view captures of moving humans, which we added to improve generalization to real videos, since MultiCamVideo is synthetic. Despite the additional sources, our training set is smaller than Vista4D's. Vista4D trains on a 60K-clip subset of OpenVid-HD together with MultiCamVideo, whereas we use 14.9K OpenVid-HD clips and 22.9K real monocular clips in total (OpenVid-HD and DynPose), about 36K clips over all sources.
The statement in Appendix B. It was based on qualitative observations rather than a controlled ablation. For DL3DV, we use the first half of each video as the source and the second half as the target, which we later found often places the two cameras so far apart that the render is very sparse. HuMMan is captured in a fixed studio, with little motion of the subject. We therefore expected both datasets to contribute little, but as the reviewer points out, we have no quantitative evidence for this. We will remove this sentence in the revision.
W3 / Q3: Direct evidence for the trust dilemma
We add two experiments. The first shows that the baselines indeed face the trust dilemma. The second shows that our gain does not come only from initializing generation from the target-aligned render, but also from resolving this dilemma.
Probing the trust of Vista4D in the render. The attention Vista4D pays to the render is highly uneven, and this unevenness predicts its failure. Recording the attention from the target tokens to the render tokens (averaged over all 40 layers), we find that its share on the dynamic region varies by 2.1× across the 110 Vista4D-Eval clips. Clips where the model attends less to the render show clearly worse camera control: the 20 clips with the lowest share have a mean RotErr of 5.02°, versus 3.06° for the remaining clips, and all of the 10 lowest show clear 3D inconsistency of the dynamic subject. Vista4D's trust in the render is therefore uncontrolled, and low trust directly hurts controllability.
Forcing more trust. If low attention to the render is the problem, forcing more attention should help, and it does, up to a point. On these 20 clips, we multiply the attention from the target tokens to the render tokens by a constant weight in every layer and step, and renormalize (Table R2). A larger weight steadily improves camera control (RotErr 6.15° → 4.30°), but it also steadily degrades visual quality (AQ 59.4 → 57.4). Even at the largest weight, the boosted Vista4D remains clearly behind ours on the same 20 clips (RotErr 4.30° versus 3.02°, and target-fitted TransErr 0.092 versus 0.080).
This remedy is, moreover, neither learned nor universal: it is a manual inference-time bias, and it does not hold across scenes (RotErr gets worse on 4–8 of the 20 clips). Too little trust leaves the dynamic subject inconsistent; forcing more trust degrades the video. No single level of trust works for every scene. This is the trust dilemma that a model conditioned on the render has to resolve, and our model avoids it by starting generation from the render instead of attending to it.
Table R2. Forcing Vista4D to attend more to the render (Vista4D-Eval, the 20 clips with the lowest render attention, VGGT-Omega point clouds, seed 42).
| Setting | RotErr ↓ | TransErr ↓ (target-aligned) | TransErr ↓ (source-fitted) | AQ ↑ | IQ ↑ |
| Vista4D | 6.15 | 0.041 | 0.267 | 59.4 | 67.2 |
| Attention weight ×1.5 | 5.99 | 0.031 | 0.248 | 59.0 | 66.8 |
| Attention weight ×2 | 5.53 | 0.027 | 0.247 | 57.9 | 66.7 |
| Attention weight ×4 | 4.30 | 0.024 | 0.239 | 57.4 | 65.7 |
Disentangling the initialization from the persistent render. We agree that starting from the target-aligned render by itself helps preserve geometry, and it is one of the two parts of our design. To measure how much of our gain comes from it, we train the control suggested by the reviewer: the GEN3C-style variant, which concatenates the render along the channel axis at every step, additionally initialized from our geometric noise. Compared with the GEN3C-style variant, it isolates the gain from the initialization; compared with ours, the gain from removing the render as a persistent input. All variants share the same source-video stream, training data and training budget.
Table R3. Disentangling the render-based initialization from the persistent render (Vista4D-Eval, 103 clips, 50 sampling steps, seed 10027; all variants trained for the same 15k steps).
| Variant | Render-based initialization | Persistent render input | RotErr ↓ | TransErr ↓ (target-aligned) | TransErr ↓ (source-fitted) | AQ ↑ | IQ ↑ |
| GEN3C-style (channel concat) | ✗ | ✓ | 4.96 | 0.044 | 0.448 | 54.7 | 69.0 |
| Channel concat + render-based initialization | ✓ | ✓ | 3.90 | 0.038 | 0.480 | 52.8 | 68.2 |
| Ours | ✓ | ✗ | 3.10 | 0.033 | 0.391 | 53.7 | 69.6 |
Initializing from the render clearly improves the GEN3C-style variant, but a gap to ours remains when the render is kept as a persistent input. The gain of our method therefore comes from both aspects: starting from the geometry, and removing the persistent render so that the network does not have to weigh it against the source video. The failure modes in the paper are consistent with this conflict: the GEN3C-style variant over-relies on the render and leaves dynamic objects fragmented, with 1.6× our rotation error, whereas Vista4D under-trusts the render on dynamic subjects, leading to 3D inconsistency (Figs. 4, 6 and 8; Table R2).
Our design therefore also does not follow the render blindly: when the dynamic mask is removed and the render of the moving subject is corrupted, it still recovers the correct motion from the source video (Fig. 7). We will add it to the architecture ablation.
Thanks again for the comments — our response is posted above, and we would be happy to address any remaining point you have. We look forward to hearing from you.
W1 / Q1: Dependence on the training geometry; budget of the architecture ablation
We thank the reviewer for these points; below we vary the input geometry as well and clarify the training budget of the architecture ablation.
Input geometry from other reconstructors. We agree that changing only the pose evaluator does not address this confounder; we therefore additionally vary the input geometry. The full study is given in our general response (“General Response to All Reviewers: Dependence on VGGT-Omega and consistency across pose estimators”): we condition every method on point clouds reconstructed with Pi3 and Depth-Anything-3 (DA3), and evaluate each output with VGGT-Omega, Pi3 and DA3, giving 3 × 3 settings in total (Table G1 there). In short, our rotation error is the lowest in all 9 settings, and with the well-posed target-aligned protocol our translation error is the lowest in all 9 settings. In particular, with Vista4D's own Pi3 geometry, our rotation error is 2.84° versus 3.44° for Vista4D under VGGT-Omega evaluation, and 2.83° versus 3.44° under Pi3 evaluation. Therefore, although only VGGT-Omega is used for training and evaluation, Manifold4D still generalizes well, and none of the mild self-consistency effects observed across evaluators changes the final ranking.
Below we show only the most important numbers; the full analysis and the complete 3 × 3 results, including the source-fitted scale, are given in our general response.
Table R1. Camera control with different input geometries and pose evaluators (Vista4D-Eval, 103 clips, seed 42). Each cell lists GEN3C / Vista4D / Ours.
Training budget of the architecture ablation. We ended the architecture ablation once the metrics had nearly plateaued. Beyond 15k steps, TransErr no longer improves and RotErr improves by only 0.2–0.4°. Among the variants, our design departs the most from the pretrained prior, since the render is injected into the starting noise rather than given as an additional condition. Even so, it has essentially converged by 15k steps, so we consider this budget sufficient for the comparison. Fully training one 14B variant takes about 2,400–3,600 A100 GPU-hours, so training every variant to the full schedule was beyond our budget.