acceptodds
Under review as a conference paper at ICLR 2027

Proxy2World: Learning to Generate Worlds from Lightweight Proxies without Seeing Them

Abstract

Lightweight scene proxies let creators control scene layout and motion while leaving room for imagination in appearance, lighting, and visual effects. However, a suitable proxy is not uniquely defined, making paired proxy–video data difficult to construct automatically at scale. We present Proxy2World, a controllable world model that learns these complementary capabilities from ordinary posed RGBD videos, without training on authored proxy–video pairs. The model jointly learns depth-conditioned RGB generation and joint RGBD generation through cross-modal flow matching. Learning both tasks enables proxy–camera hybrid denoising at inference to follow the proxy structure while producing natural, detailed visuals. We further introduce ProxyBench to evaluate this capability across a diverse set of scenes, camera trajectories, and subject motions. Experiments on ProxyBench show that Proxy2World achieves a better balance between structural adherence and visual quality than camera-controlled and geometry-conditioned methods, supported by quantitative metrics, VLM assessments, human evaluations and diverse qualitative results.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Controllable video generation, Generative rendering, Scene Proxies

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 4 reviews, above 99% of the others.
Reviewer_aUjPOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper presents Proxy2World, a method that generates camera-controlled videos following the layout of a coarse user-authored 3D proxy, without paired proxy–video data. Wan2.2-I2V is fine-tuned on about 120k posed RGB-D clips with separate noise levels for RGB and depth, so one model learns both depth-conditioned RGB generation and joint RGB-D generation. At sampling time, the proxy depth is imposed as a clean condition for the first steps and then released into the joint mode; the default τ* = 0.925 keeps the proxy for 3 of 40 steps. The authors also present ProxyBench and compare against camera-, depth- and proxy-conditioned baselines using VIPE camera errors, alignment metrics, Gemini win rates and a small human study.

Critical strengths

Learning from ordinary posed RGB-D video avoids the main data bottleneck of proxy-to-video rendering, and the qualitative results transfer to box, primitive and incomplete proxies.

One checkpoint covers a range of control strengths, and the internal evidence is clean. Within the same model and inputs, moving from camera-only TI2V to mixed TI2V raises Geo Align from 0.534 to 0.680, VLM adherence from 53.55 to 64.18 and refinement from 71.44 to 77.96, and the overall rank in the human study improves from 5 to 1. The paper also reports the trade-off honestly: depth-only methods win on alignment, the mixed setting wins on refinement, and the human-preference counts reconcile (270 + 252 + 240 = 762).

Critical weaknesses

The main comparison uses different input conditions. "Ours (TI2V, τ*)" uses a reference first frame, while VACE-Depth and Cosmos-Depth run without one. Without the reference frame, "Ours (T2V, τ*)" reaches only 42.81 VLM adherence against 60.14 for VACE-Depth and 72.26 for Cosmos-Depth, barely above Coarse2Real (40.03). Even with the reference frame, mixed TI2V trails Cosmos-Depth on adherence (64.18 vs 72.26) and Geo Align (0.680 vs 0.771), and in the human study VACE-Depth has the highest adherence preference (81.8%). The headline comparison is therefore confounded by the reference image.

The rotation metric can reward incorrect motion. Eq. 22 normalises orientation error by max(Φ̂, 10°), where Φ̂ is the accumulated rotation of the evaluated method's own trajectory. A prediction that is correct at the first frame and then holds a constant 10° offset, and one that alternates ±20°, have similar mean orientation error but nATEr of about 0.98 and 0.003 respectively, so jitter makes a method look better. Arc-length resampling also removes timing errors altogether. The "77.2% rotation-error reduction" relies on this score, so raw ATE and target-normalised, frame-aligned errors are needed.

The straightforward baseline of early-step control in existing depth-conditioned models is missing. Applying VACE-Depth or Cosmos-Transfer with proxy depth for the first k steps, with k matched to τ*, would test whether joint RGB-D training is actually needed. In addition, the proxy acts as a clean condition for only 3 of 40 steps, there is no quantitative τ curve and no breakdown by proxy abstraction level, and the evaluation uses 78 clips from 5 of 25 worlds selected for "rich interaction designs", with a 10-participant interim human study in which the two top-ranked variants were compared directly only 5 times. The ingredients themselves are known (per-modality timesteps in a shared transformer, joint geometry–appearance generation, early-step structure guidance as in SDEdit or ControlNet guidance-end).

Questions for authors
  1. Can VACE-Depth be run with the same reference first frame, or can the reference-free T2V setting be made the headline comparison?
  2. Do the camera-control conclusions hold with raw ATE and with target-normalised, frame-aligned errors?
  3. How do VACE-Depth and Cosmos-Transfer perform when proxy depth is applied only for the first k steps, with k matched to τ*?
5 comments

We thank the reviewer for recognizing the value of learning proxy-controlled video generation without paired proxy-video data and the control-strength flexibility of one checkpoint. We agree that the reference-image setting, camera metrics, and the alternative of early-only depth guidance need clearer evidence. We address these three points below, adding quantitative comparisons of control duration, late refinement, switching ratio, and proxy abstraction.

Q1: Fair comparison and the adherence-refinement balance

Under the same reference-free setting, our method better refines coarse proxy geometry and appearance while largely preserving the intended layout, achieving a more favorable adherence–refinement trade-off.

We compare all methods in the same reference-free setting and report TI2V separately. This is our primary input-matched comparison; TI2V with camera and proxy inputs remains our main authoring setting. We assess adherence and refinement together because reproducing a coarse proxy faithfully does not necessarily yield natural geometry and appearance.

Table 1. Reference-free comparison. Scores are percentages except rank; user-study scores are aggregate preference rates.

MethodVLM AdherenceVLM RefinementUser Study AdherenceUser Study RefinementUser Study Overall preferenceUser Study Overall rank
VACE-Depth60.1445.3281.844.065.94
Cosmos-Depth72.2640.2872.261.171.43
Coarse2Real40.0345.2143.133.027.08
Ours T2V, mixed42.8151.3075.088.282.32

VACE-Depth and Cosmos-Depth adhere more closely to the proxy, but this does not necessarily translate into better refinement of its coarse geometry. Without a reference image, Ours achieves the highest VLM refinement (51.30%), human refinement preference (88.2%), and overall preference (82.3%) among the compared methods. The human study also shows that Ours retains substantial adherence (75.0%, versus 81.8% for VACE-Depth). Together, these results support our intended balance: preserving the proxy’s main layout while producing more natural geometry and appearance.

Q2: Camera control with corrected metrics

Camera-control error should measure deviation from the commanded trajectory at the commanded times, using a normalization that cannot be improved by adding motion to the prediction.

We agree that a prediction-dependent denominator can reward excessive camera motion, and that arc-length resampling can hide timing errors. We corrected both issues and recomputed all trajectories on the original 81 frame timestamps, ti=i/16t_i=i/16, without time warping. Every method for a given clip is now normalized by the same target trajectory, so predicted jitter cannot inflate the denominator and improve the score.

Both trajectories are expressed relative to their own first camera. Rotation errors eire_i^r are SO(3)\mathrm{SO}(3) geodesic angles in degrees. Translation errors are eit=∥s∗c^i−ci∥2e_i^t = \left\lVert s^{\ast}\widehat{\mathbf{c}}_i - \mathbf{c}_i \right\rVert_2, where one nonnegative scale s∗s^{\ast} aligns reconstruction scale. We report rotation MAE/RMSE and translation RMSE, together with target-normalized errors:

nRMAE=1N∑i=1Neirmax⁡!(Atarget,,10∘).\mathrm{nR}_{\mathrm{MAE}} = \frac{\frac{1}{N}\sum_{i=1}^{N} e_i^r} {\max\\!\left(A_{\mathrm{target}},\\,10^{\circ}\right)}.

nTRMSE=1N∑i=1N(eit)2max⁡!(Ltarget,,0.5).\mathrm{nT}_{\mathrm{RMSE}} = \frac{\sqrt{\frac{1}{N}\sum_{i=1}^{N}\left(e_i^t\right)^2}} {\max\\!\left(L_{\mathrm{target}},\\,0.5\right)}.

Here N=81N=81, AtargetA_{\mathrm{target}} is cumulative target rotation, and LtargetL_{\mathrm{target}} is target path length. The floors stabilize near-stationary targets.

Table 2. Corrected camera errors. Bold marks the best result within each control group. Translation RMSE uses target-camera units after scale alignment; normalized errors are dimensionless.

ConfigurationR MAE (°) ↓R RMSE (°) ↓T RMSE ↓nRMAE\mathrm{nR}_{\mathrm{MAE}} ↓nTRMSE\mathrm{nT}_{\mathrm{RMSE}} ↓
Camera-controlled
SCoPE8.394310.25660.49630.14010.1143
Lyra23.06374.83500.28060.02360.0428
FantasyWorld17.580421.47940.90960.22450.2501
Ours TI2V, camera-only3.05703.55850.28730.04220.0696
Depth-controlled
Cosmos-Depth1.68811.94750.18260.03070.0447
VACE-Depth3.09683.70870.29810.04890.0763
Ours T2V, full-depth1.19091.39300.13230.02140.0285
Proxy-controlled
C2R14.965917.95440.69910.14220.1503
Ours T2V, mixed3.82924.37750.33330.05820.0856
Ours TI2V, mixed2.81883.31690.26360.04170.0666

The corrected protocol changes the numerical scores and some rankings, and we replace the earlier ``77.2% reduction'' statement with these results. Our method retains leading results within each control group: the lowest rotation MAE/RMSE for camera-controlled generation, and the lowest errors on all five metrics for depth-controlled and proxy-controlled generation. Compared with full-depth control, mixed proxy control gives up some camera precision to allow geometry and appearance refinement, consistent with the intended control-refinement trade-off.

Q3: Can early-only depth control replace joint refinement?

Releasing proxy depth creates freedom to refine geometry, but the subsequent denoising process must retain structural guidance; our model does this by jointly evolving RGB and depth under persistent camera conditioning.

Early release alone does not reproduce our adherence-refinement balance in the tested comparison. Following the reviewer's suggestion, we apply VACE depth control during the first k∈{7,15,23,50}k\in\{7,15,23,50\} of 50 denoising steps, then disable its control residuals. The tested clips share the same proxy, caption, and seed across VACE settings; all methods are reference-free. The 15/50 cutoff matches our default 3/40 release noise, σ=0.973684\sigma=0.973684.

Table 3. Depth-release comparison. VLM win rates (%); camera errors from VGGT-Omega.

SetupControlled VACE stepsAdherenceRefinementRotation MAE (degrees)Target-normalized translation
VACE-Depth7/5034.3840.6332.070.2994
VACE-Depth15/5026.5632.8121.480.1872
VACE-Depth23/5032.8131.257.470.1156
VACE-Depth50/5079.6970.312.100.0189
Ours T2V, tau*--76.5675.003.580.0389

Visually, early release can reduce the proxy-like character shapes, but it also changes the intended passage, railing, and building layout (Figure S3). These changes are therefore not limited to refining appearance while preserving structure. Table 3 shows lower adherence and refinement than full-depth VACE, together with larger camera errors. At the matched cutoff, VACE obtains 26.56% adherence and 32.81% refinement, compared with 76.56% and 75.00% for Ours.

We interpret this as a limitation of the post-release process: VACE-Depth generates RGB under depth conditioning, but disabling that conditioning does not give it a geometry-preserving joint refinement stage. Later denoising is not automatically restricted to improving coarse shapes while keeping their layout.

To test whether joint continuation is essential within our own model, we perform an analogous ablation: after the identical three-step grounding stage, we remove the Depth token sequence and continue camera-conditioned RGB-only refinement. We retain the exact RGB state at the switch, camera conditions, and model weights. This tests whether releasing depth while retaining camera guidance is sufficient, or whether generating geometry alongside appearance provides an additional benefit.

Table 4. Effect of late joint RGBD refinement on layout alignment.

SettingLate continuationGeo Align (higher is better)
T2VRGB-only0.2945
T2VJoint RGBD0.5048
TI2VRGB-only0.3214
TI2VJoint RGBD0.4883

Effect of late joint RGBD refinement on layout alignment.

Removing joint continuation lowers Geo Align from 0.5048 to 0.2945 in T2V and from 0.4883 to 0.3214 in TI2V, with a decrease on every paired clip. Thus, camera guidance alone after depth release does not retain the same structural control. Figure S2 also shows scene and character refinement in native joint-generated depth.

The key distinction is the denoising process after proxy release. Our model learns depth-conditioned RGB generation and camera-guided joint RGBD generation, with metric alignment keeping depth and camera translations in a common physical scale. After grounding, camera Grid remains a clean condition while RGB and depth are jointly denoised. Our balance comes from coordinating proxy-depth grounding with this continuing camera-guided geometry-appearance refinement; it cannot be recovered simply by removing depth control early from an otherwise depth-conditioned generator.

Reviewer_sBoBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Proxy2World, which is a proxy-conditioned bidirectional video generation model. It is camera-conditioned and jointly generates RGB and depth videos. To solve the problem of scarcity of proxy-video pair data, the paper uses a hybrid denoising method, where clean proxy depth is provided in the first few steps of the denoising process, and then depth and RGB are jointly generated from the same noise level. In this way, the model can only be trained with fine-grained depth and applied to the proxy-depth condition.

The paper proposes a ProxyBench with 25 proxy scenes. On the benchmark, the Proxy2World model outperforms other proxy-based and camera-conditioned baselines.

Critical strengths
  1. The hybrid denoising method is a promising solution when proxy-video pair data is hard to scale. Fine-grained depth is much easier to obtain from off-the-shelf models. It’s a practical method to address the data scarcity of code-to-world generation.
Critical weaknesses
  1. The experiments regarding the hybrid denoising process are not solid:

i) Figure 5(a) is the only qualitative comparison for the hybrid method, with only one frame of each video shown. From this figure, I feel the proxy-only variant has better background scene generation quality, and the mix signal is only better in character generation.

ii) The results in Table 1 cannot fully support the effectiveness of the hybrid denoising. In the T2V version, hybrid is only better in Reproj., Ref. WR and human rank (how to explain that spatial consistency is better without the depth condition). In the TI2V version, the improvement against \tau=1 is also marginal. More explanation is required here.

iii) No ablation on the choice of \tau. I would prefer a performance curve sweeping \tau from 0 to 1 rather than results at a single value. Side-by-side video comparison would also be great.

Questions for authors
  1. In Figure 1 and the provided project page, the proxy building and interactive control functions are demonstrated. However, the paper provides no details on the agent framework for proxy building and interactive control. Are they part of the paper?

  2. Clarify the difference between the T2V and TI2V variants. Which one should be deemed the main model? In Table 1, what is the performance of \tau=1 with T2V and \tau=0 with TI2V?

  3. See weaknesses about the hybrid denoising experiment.

4 comments

We thank the reviewer for recognizing the potential of hybrid denoising for learning proxy-controlled generation without paired proxy–video training data. We clarify the interpretation of Main Table, our intended input setting, and the role of the authoring workflow below.

Q1: Table 1 results regarding hybrid denoising

Hybrid denoising uses proxy depth to establish structure and camera-guided joint RGBD generation to refine it; its objective is to balance layout control with visual quality, rather than maximize proxy fidelity alone.

Our goal is to combine camera control and proxy guidance through joint RGBD denoising, preserving the intended layout and motion while refining coarse geometry and appearance. The table below shows this adherence–refinement balance within each reference-image setting.

In T2V, full-depth conditioning has higher proxy alignment and adherence, while mixed sampling improves refinement from 31.71 to 51.30 and overall human rank from 6 to 2. This is the intended trade-off: reproducing the coarse proxy more closely does not necessarily produce a more natural or detailed video. In TI2V, mixed sampling improves both control and refinement over camera-only generation: Geo Align increases from 0.5340 to 0.6795, adherence from 53.55 to 64.18, and refinement from 71.44 to 77.96; overall human rank improves from 5 to 1. These results support the value of proxy grounding for layout preservation while retaining visual refinement.

Table 1. Hybrid sampling within each reference-image setting.

ConfigurationReproj. ↓Geo. ↑Temp. ↑Adh. ↑Ref. ↑Rank ↓
T2V full depth2.2800.77490.764568.7931.716
T2V mixed2.0670.70840.712642.8151.302
TI2V camera only1.9560.53400.703353.5571.445
TI2V mixed2.5240.67950.717364.1877.961

The reprojection metric measures the consistency of a reconstruction fitted to the generated video, rather than agreement with the prescribed proxy. Thus, lower reprojection error can coexist with lower proxy adherence. Mixed sampling is also not generation without depth conditioning: the proxy establishes structure during the first three steps, followed by 37 steps of joint RGBD refinement with camera conditioning maintained. This explains why releasing the fixed depth constraint need not eliminate spatial consistency. The results support a favorable balance between layout control and visual quality, rather than the best value on every metric.

Q2: Which is the main setting, and how do T2V and TI2V differ?

TI2V with camera and proxy inputs is our intended setting: the first RGB frame controls appearance, the camera controls viewpoint motion, and the proxy guides scene layout and object motion.

Our main setting is TI2V with camera and proxy inputs. T2V uses the same model and differs only in omitting the first RGB frame. TI2V provides the fullest control: the image specifies appearance and detail, while camera and proxy inputs guide viewpoint and layout. Our pipeline also supports generating a proxy-aligned first frame. This is the intended setting for our joint RGBD denoising framework combining camera control and proxy guidance.

T2V at τ=1\tau=1 is camera-only generation without a reference image; TI2V at τ=0\tau=0 adds a reference image to full-depth conditioning. Our main comparisons match the evaluated baselines' inputs: reference-conditioned camera methods and reference-free depth methods. We will include videos of both additional endpoints in the supplementary material.

Q3: What is the role of the agent and interactive-control pipeline?

The agentic workflow turns authoring intent into the whitebox, scripted interactions, and aligned first frame that serve as inputs to our video generator.

The agentic component is an authoring workflow for constructing whitebox scenes, adding script-driven mechanisms and interactions, and producing an aligned first RGB frame. Users can edit the scene and camera before rendering the controls for video generation. We will describe this pipeline in the supplementary material and release it together with ProxyBench. It supports input creation and interactive authoring; the paper's central methodological contribution remains the joint RGBD video generator and its combination of camera control and proxy guidance.

Q4: Qualitative comparisons across switching ratios

The switching-ratio sweep table (Figure S4) and synchronized comparisons are provided in the supplementary material (Video S1). We include τ=0,0.7,0.925\tau=0, 0.7, 0.925 (default), and 11, keeping the other inputs fixed within each clip to illustrate the balance between proxy adherence and refinement.

Reviewer_nzhaOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Proxy2World generates videos from simple 3D scene proxies while aiming to preserve their layout and motion. It learns two tasks from posed RGBD videos, without paired proxy–video examples: generating RGB from depth, and generating RGB and depth together. During sampling, proxy depth first establishes the scene structure, then both appearance and depth are refined together. ProxyBench results suggest that this produces more natural shapes than strict depth conditioning while retaining more layout control than camera-only generation.

Critical strengths

The insight is to treat a rough proxy as a guide rather than a shape the output must copy. This addresses a practical need: users can specify where objects are and how they move without modeling every detail. Learning both generation modes from ordinary RGBD video avoids collecting proxy–video training pairs, and one model supports different control strengths. The comparisons in Figure 5 support this design, while the shared metric scale keeps depth and camera guidance consistent.

Critical weaknesses
  1. The role of joint depth generation needs clearer evidence. Section 4.3 compares it with RGB-only refinement and reports better camera accuracy and human preference. However, it does not show whether the generated depth becomes more plausible while retaining the intended layout. This leaves uncertainty about the central claim: improving scene geometry, rather than simply making the images look better.

  2. The claim of handling different levels of proxy detail is also not well established. The same switching ratio is used throughout, but the paper does not systematically test progressively simpler versions of the same scene. Table 1 supports a useful compromise between control and realism, but does not establish a better trade-off than the alternatives. The strongest variant receives an extra reference image, and the baselines are not compared across control strengths.

Questions for authors

Can you show how joint refinement improves geometry beyond RGB-only refinement? How sensitive is the switching ratio to simpler proxies? With the same inputs and comparable control strength, does the method still offer a better balance between layout preservation and visual quality?

3 comments

We thank the reviewer for recognizing the practical value of treating rough proxies as guides, learning both generation modes from ordinary RGBD videos, and keeping depth and camera guidance in a shared metric scale. We sincerely agree that clearer evidence is needed on whether joint generation refines scene geometry and whether a fixed switching ratio retains our method's advantage across proxy abstraction levels. We address these points and the fairness of the baseline comparison below.

Q1. Does joint refinement improve scene geometry?

Joint generation treats depth as an evolving scene representation, allowing geometry and appearance to be refined together while preserving the structure established by proxy grounding.

Quantitatively, we use Geo Align to assess how well the geometry of generated RGB videos preserves the proxy's intended spatial layout. We compare joint RGBD and RGB-only refinement after an identical proxy-grounding stage, keeping the remaining inputs fixed and applying the same evaluation to both outputs. This isolates the effect of generating depth together with RGB during refinement.

Table 1 shows that joint RGBD refinement better preserves the intended layout in both T2V and TI2V. In Figure S2, the native joint-generated depth is also refined: simplified proxy actors become more articulated human silhouettes, and regular scene surfaces acquire richer contours and surface structure, while the main walls, columns, railings, and traversable routes remain recognizable. These results demonstrate our joint generation model's ability to refine scene and character geometry alongside RGB appearance while preserving the intended spatial arrangement.

Table 1. Coarse layout alignment after refinement.

SettingLate continuationGeo Align (higher is better)
T2VRGB-only0.2945
T2VJoint RGBD0.5048
TI2VRGB-only0.3214
TI2VJoint RGBD0.4883

Q2. Does switching ratio sensitive across proxy levels?

Our method is robust to different proxy abstraction levels: the same switching ratio achieves higher refinement than the compared baselines across all three levels while retaining competitive layout adherence.

To examine whether the switching ratio needs to be adjusted for different proxy abstraction levels, we sweep tau over original authored, box-primitive, and incomplete versions of the same scenes while keeping the other inputs fixed. Table 2 shows that concentrating proxy guidance in the early high-noise stage is consistently effective across all three levels. The same tau* achieves the highest refinement while retaining competitive adherence. Thus, the preferred ratio has limited sensitivity to proxy abstraction: the same high-noise guidance strategy gives a favorable balance across all three proxy levels without proxy-specific retuning.

Table 2. Adherence and refinement across proxy conditions (%).

ktauOriginal authored: Adh.Ref.Box primitives: Adh.Ref.Incomplete: Adh.Ref.
01.00025.0029.1737.5045.8320.8341.67
30.92556.2568.7562.5089.5864.5883.33
120.70058.3352.0850.0043.7552.0841.67
400.00060.4250.0050.0020.8362.5033.33

Using the best-performing tau* from this sweep, we compare our method with Coarse2Real and VACE-Depth across the same three proxy levels. All methods, for a fair comparison, generate without a reference RGB image. Table 3 shows that our method achieves the highest refinement at every proxy level, while VACE-Depth has higher adherence and our method improves both criteria over Coarse2Real. Figure S1 also illustrates this trade-off: our outputs retain the main layout while refining coarse shapes, VACE-Depth retains more proxy-like geometry, and Coarse2Real departs further from the specified layout. The same tau* therefore retains a favorable adherence-refinement balance against both baselines without reference-image conditioning.

Table 3. Adherence and refinement across proxy levels (%).

MethodLow-poly Adh.Ref.Box primitives Adh.Ref.Incomplete Adh.Ref.
Coarse2Real15.6331.2525.0046.88‾\underline{46.88}31.2543.75
VACE-Depth68.7543.75‾\underline{43.75}68.7525.0068.7550.00‾\underline{50.00}
Ours65.63‾\underline{65.63}75.0056.25‾\underline{56.25}78.1350.00‾\underline{50.00}56.25

Thank you for your thoughtful response. The additional experiments further strengthen the paper, and I will raise my score to accept. Good luck.

Reviewer_kPduOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a training framework for generating videos conditioned on 3D proxies using ordinary posed RGBD videos, without requiring paired proxy–video training data. It uses a shared flow-matching model for both depth-conditioned RGB generation and joint RGBD generation. During inference, the model first follows the rendered proxy depth to establish the scene structure, then gradually refines the geometry and appearance together. Metric-scale alignment helps keep the depth and camera conditioning consistent. It also introduce ProxyBench to evaluate how well the generated videos follow the proxy, improve upon its geometry, and respect camera controls.

Critical strengths
  1. Treating depth as both a conditioning signal and a generated variable allows these two capabilities to be learned from ordinary posed RGBD videos. This enables transfer to authored proxies without collecting matched coarse-proxy and realistic-video pairs, addressing a practical bottleneck in training proxy-controlled generators. The shared training framework is also a clever design, enabling both depth-conditioned RGB generation and joint RGBD generation within a single model.
  2. The staged sampler try to addresses the central tension in proxy-based generation: preserving intended layout and motion while improving coarse geometry. Early depth conditioning establishes structural grounding; subsequent joint RGBD generation refines geometry and appearance together while maintaining camera guidance.
Critical weaknesses

The relationship to Modality Forcing [1] is not discussed. MoFo already unifies conditional and joint RGB–depth generation using modality-specific noise levels and adjusts depth-conditioning strength through denoising schedules. The paper should clarify the additional contribution of its staged proxy refinement, particularly how freely updating depth differs from MoFo’s partial-depth conditioning.

[1] Duisterhof et al. Modality Forcing for Scalable Spatial Generation, 2026.

Questions for authors

With the scene, camera, and reference fixed, how do adherence and refinement vary across low-poly, primitive, and incomplete proxies?

1 comment

We thank the reviewer for recognizing the practical benefit of learning from ordinary posed RGBD videos without paired proxy-video data, the shared framework for conditional and joint generation, and the staged sampler's design for structural grounding and joint geometry and appearance refinement. We agree that experiments across different levels of proxy abstraction are needed to better understand the robustness and behavior of our approach.

Q1. Robustness to Different Levels of Proxy Abstraction

Our model handles proxies at different levels of abstraction, preserving their intended layout while refining coarse geometry and completing missing structure.

Difference to MoFo. MoFo[1] is a joint image-depth generation model built on text-to-image backbones. Its partial-depth conditioning controls how strongly an image follows a supplied depth map. Our focus is the coarseness of the conditioning signal for video generation from whiteboxes built by agents or humans: the model learns from detailed depth while accepting abstract or incomplete proxies at inference.

Experimental Design. Following the reviewer's suggestion, we evaluate low-poly, primitive, and incomplete proxies. Low-poly uses the original authored coarse whitebox; primitive replaces compound structures and actor parts with bounding boxes; incomplete removes visible structural groups while retaining the actor and ground. Within each method, we keep every other condition fixed and vary only the proxy video. All methods generate without a reference RGB image. Adherence and refinement use the clarified VLM rubric; original prompts and scores are retained in the source bundle.

Table 1. Adherence/refinement win rates (%). Best in bold; second best underlined.

MethodLow-poly Adherence (%)Low-poly Refinement (%)Primitive Adherence (%)Primitive Refinement (%)Incomplete Adherence (%)Incomplete Refinement (%)
C2R15.6331.2525.0046.88‾\underline{46.88}31.2543.75
VACE-Depth68.7543.75‾\underline{43.75}68.7525.0068.7550.00‾\underline{50.00}
Ours65.63‾\underline{65.63}75.0056.25‾\underline{56.25}78.1350.00‾\underline{50.00}56.25

The same trade-off holds across all three levels of proxy abstraction: VACE-Depth achieves higher adherence, while Ours achieves higher refinement. Figure S1 and table 1 illustrates how our method adapts to the proxy. For low-poly proxies, it follows the coarse geometry and preserves semantic cues when the shapes are informative. For primitive proxies, it preserves the layout while imagining more detailed semantics and appearance. For incomplete proxies, it follows semantic and spatial cues in the specified regions and plausibly completes the remaining areas. VACE-Depth closely fits the proxy but retains coarse, proxy-like geometry and a rougher appearance. C2R produces realistic-looking imagery but substantially deviates from the specified layout. Together with the main table, these observations reinforce the paper's central conclusion: hybrid denoising balances layout control with visual quality across different levels of proxy abstraction.

[1] Duisterhof et al. Modality Forcing for Scalable Spatial Generation. arXiv:2606.13676, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.