- Rating
- 4(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents Proxy2World, a method that generates camera-controlled videos following the layout of a coarse user-authored 3D proxy, without paired proxy–video data. Wan2.2-I2V is fine-tuned on about 120k posed RGB-D clips with separate noise levels for RGB and depth, so one model learns both depth-conditioned RGB generation and joint RGB-D generation. At sampling time, the proxy depth is imposed as a clean condition for the first steps and then released into the joint mode; the default τ* = 0.925 keeps the proxy for 3 of 40 steps. The authors also present ProxyBench and compare against camera-, depth- and proxy-conditioned baselines using VIPE camera errors, alignment metrics, Gemini win rates and a small human study.
Critical strengths
Learning from ordinary posed RGB-D video avoids the main data bottleneck of proxy-to-video rendering, and the qualitative results transfer to box, primitive and incomplete proxies.
One checkpoint covers a range of control strengths, and the internal evidence is clean. Within the same model and inputs, moving from camera-only TI2V to mixed TI2V raises Geo Align from 0.534 to 0.680, VLM adherence from 53.55 to 64.18 and refinement from 71.44 to 77.96, and the overall rank in the human study improves from 5 to 1. The paper also reports the trade-off honestly: depth-only methods win on alignment, the mixed setting wins on refinement, and the human-preference counts reconcile (270 + 252 + 240 = 762).
Critical weaknesses
The main comparison uses different input conditions. "Ours (TI2V, τ*)" uses a reference first frame, while VACE-Depth and Cosmos-Depth run without one. Without the reference frame, "Ours (T2V, τ*)" reaches only 42.81 VLM adherence against 60.14 for VACE-Depth and 72.26 for Cosmos-Depth, barely above Coarse2Real (40.03). Even with the reference frame, mixed TI2V trails Cosmos-Depth on adherence (64.18 vs 72.26) and Geo Align (0.680 vs 0.771), and in the human study VACE-Depth has the highest adherence preference (81.8%). The headline comparison is therefore confounded by the reference image.
The rotation metric can reward incorrect motion. Eq. 22 normalises orientation error by max(Φ̂, 10°), where Φ̂ is the accumulated rotation of the evaluated method's own trajectory. A prediction that is correct at the first frame and then holds a constant 10° offset, and one that alternates ±20°, have similar mean orientation error but nATEr of about 0.98 and 0.003 respectively, so jitter makes a method look better. Arc-length resampling also removes timing errors altogether. The "77.2% rotation-error reduction" relies on this score, so raw ATE and target-normalised, frame-aligned errors are needed.
The straightforward baseline of early-step control in existing depth-conditioned models is missing. Applying VACE-Depth or Cosmos-Transfer with proxy depth for the first k steps, with k matched to τ*, would test whether joint RGB-D training is actually needed. In addition, the proxy acts as a clean condition for only 3 of 40 steps, there is no quantitative τ curve and no breakdown by proxy abstraction level, and the evaluation uses 78 clips from 5 of 25 worlds selected for "rich interaction designs", with a 10-participant interim human study in which the two top-ranked variants were compared directly only 5 times. The ingredients themselves are known (per-modality timesteps in a shared transformer, joint geometry–appearance generation, early-step structure guidance as in SDEdit or ControlNet guidance-end).
Questions for authors
- Can VACE-Depth be run with the same reference first frame, or can the reference-free T2V setting be made the headline comparison?
- Do the camera-control conclusions hold with raw ATE and with target-normalised, frame-aligned errors?
- How do VACE-Depth and Cosmos-Transfer perform when proxy depth is applied only for the first k steps, with k matched to τ*?
5 comments
Q2: Camera control with corrected metrics
Camera-control error should measure deviation from the commanded trajectory at the commanded times, using a normalization that cannot be improved by adding motion to the prediction.
We agree that a prediction-dependent denominator can reward excessive camera motion, and that arc-length resampling can hide timing errors. We corrected both issues and recomputed all trajectories on the original 81 frame timestamps, , without time warping. Every method for a given clip is now normalized by the same target trajectory, so predicted jitter cannot inflate the denominator and improve the score.
Both trajectories are expressed relative to their own first camera. Rotation errors are geodesic angles in degrees. Translation errors are , where one nonnegative scale aligns reconstruction scale. We report rotation MAE/RMSE and translation RMSE, together with target-normalized errors:
Here , is cumulative target rotation, and is target path length. The floors stabilize near-stationary targets.
Table 2. Corrected camera errors. Bold marks the best result within each control group. Translation RMSE uses target-camera units after scale alignment; normalized errors are dimensionless.
| Configuration | R MAE (°) ↓ | R RMSE (°) ↓ | T RMSE ↓ | ↓ | ↓ |
|---|---|---|---|---|---|
| Camera-controlled | |||||
| SCoPE | 8.3943 | 10.2566 | 0.4963 | 0.1401 | 0.1143 |
| Lyra2 | 3.0637 | 4.8350 | 0.2806 | 0.0236 | 0.0428 |
| FantasyWorld | 17.5804 | 21.4794 | 0.9096 | 0.2245 | 0.2501 |
| Ours TI2V, camera-only | 3.0570 | 3.5585 | 0.2873 | 0.0422 | 0.0696 |
| Depth-controlled | |||||
| Cosmos-Depth | 1.6881 | 1.9475 | 0.1826 | 0.0307 | 0.0447 |
| VACE-Depth | 3.0968 | 3.7087 | 0.2981 | 0.0489 | 0.0763 |
| Ours T2V, full-depth | 1.1909 | 1.3930 | 0.1323 | 0.0214 | 0.0285 |
| Proxy-controlled | |||||
| C2R | 14.9659 | 17.9544 | 0.6991 | 0.1422 | 0.1503 |
| Ours T2V, mixed | 3.8292 | 4.3775 | 0.3333 | 0.0582 | 0.0856 |
| Ours TI2V, mixed | 2.8188 | 3.3169 | 0.2636 | 0.0417 | 0.0666 |
The corrected protocol changes the numerical scores and some rankings, and we replace the earlier ``77.2% reduction'' statement with these results. Our method retains leading results within each control group: the lowest rotation MAE/RMSE for camera-controlled generation, and the lowest errors on all five metrics for depth-controlled and proxy-controlled generation. Compared with full-depth control, mixed proxy control gives up some camera precision to allow geometry and appearance refinement, consistent with the intended control-refinement trade-off.
Q3: Can early-only depth control replace joint refinement?
Releasing proxy depth creates freedom to refine geometry, but the subsequent denoising process must retain structural guidance; our model does this by jointly evolving RGB and depth under persistent camera conditioning.
Early release alone does not reproduce our adherence-refinement balance in the tested comparison. Following the reviewer's suggestion, we apply VACE depth control during the first of 50 denoising steps, then disable its control residuals. The tested clips share the same proxy, caption, and seed across VACE settings; all methods are reference-free. The 15/50 cutoff matches our default 3/40 release noise, .
Table 3. Depth-release comparison. VLM win rates (%); camera errors from VGGT-Omega.
| Setup | Controlled VACE steps | Adherence | Refinement | Rotation MAE (degrees) | Target-normalized translation |
|---|---|---|---|---|---|
| VACE-Depth | 7/50 | 34.38 | 40.63 | 32.07 | 0.2994 |
| VACE-Depth | 15/50 | 26.56 | 32.81 | 21.48 | 0.1872 |
| VACE-Depth | 23/50 | 32.81 | 31.25 | 7.47 | 0.1156 |
| VACE-Depth | 50/50 | 79.69 | 70.31 | 2.10 | 0.0189 |
| Ours T2V, tau* | -- | 76.56 | 75.00 | 3.58 | 0.0389 |
Visually, early release can reduce the proxy-like character shapes, but it also changes the intended passage, railing, and building layout (Figure S3). These changes are therefore not limited to refining appearance while preserving structure. Table 3 shows lower adherence and refinement than full-depth VACE, together with larger camera errors. At the matched cutoff, VACE obtains 26.56% adherence and 32.81% refinement, compared with 76.56% and 75.00% for Ours.
We interpret this as a limitation of the post-release process: VACE-Depth generates RGB under depth conditioning, but disabling that conditioning does not give it a geometry-preserving joint refinement stage. Later denoising is not automatically restricted to improving coarse shapes while keeping their layout.
To test whether joint continuation is essential within our own model, we perform an analogous ablation: after the identical three-step grounding stage, we remove the Depth token sequence and continue camera-conditioned RGB-only refinement. We retain the exact RGB state at the switch, camera conditions, and model weights. This tests whether releasing depth while retaining camera guidance is sufficient, or whether generating geometry alongside appearance provides an additional benefit.
Table 4. Effect of late joint RGBD refinement on layout alignment.
| Setting | Late continuation | Geo Align (higher is better) |
|---|---|---|
| T2V | RGB-only | 0.2945 |
| T2V | Joint RGBD | 0.5048 |
| TI2V | RGB-only | 0.3214 |
| TI2V | Joint RGBD | 0.4883 |
Effect of late joint RGBD refinement on layout alignment.
Removing joint continuation lowers Geo Align from 0.5048 to 0.2945 in T2V and from 0.4883 to 0.3214 in TI2V, with a decrease on every paired clip. Thus, camera guidance alone after depth release does not retain the same structural control. Figure S2 also shows scene and character refinement in native joint-generated depth.
The key distinction is the denoising process after proxy release. Our model learns depth-conditioned RGB generation and camera-guided joint RGBD generation, with metric alignment keeping depth and camera translations in a common physical scale. After grounding, camera Grid remains a clean condition while RGB and depth are jointly denoised. Our balance comes from coordinating proxy-depth grounding with this continuing camera-guided geometry-appearance refinement; it cannot be recovered simply by removing depth control early from an otherwise depth-conditioned generator.
We thank the reviewer for recognizing the value of learning proxy-controlled video generation without paired proxy-video data and the control-strength flexibility of one checkpoint. We agree that the reference-image setting, camera metrics, and the alternative of early-only depth guidance need clearer evidence. We address these three points below, adding quantitative comparisons of control duration, late refinement, switching ratio, and proxy abstraction.
Q1: Fair comparison and the adherence-refinement balance
Under the same reference-free setting, our method better refines coarse proxy geometry and appearance while largely preserving the intended layout, achieving a more favorable adherence–refinement trade-off.
We compare all methods in the same reference-free setting and report TI2V separately. This is our primary input-matched comparison; TI2V with camera and proxy inputs remains our main authoring setting. We assess adherence and refinement together because reproducing a coarse proxy faithfully does not necessarily yield natural geometry and appearance.
Table 1. Reference-free comparison. Scores are percentages except rank; user-study scores are aggregate preference rates.
VACE-Depth and Cosmos-Depth adhere more closely to the proxy, but this does not necessarily translate into better refinement of its coarse geometry. Without a reference image, Ours achieves the highest VLM refinement (51.30%), human refinement preference (88.2%), and overall preference (82.3%) among the compared methods. The human study also shows that Ours retains substantial adherence (75.0%, versus 81.8% for VACE-Depth). Together, these results support our intended balance: preserving the proxy’s main layout while producing more natural geometry and appearance.