acceptodds
Under review as a conference paper at ICLR 2027

Code Video Model

11 views

Abstract

Video diffusion models generate realistic videos in a black box. Physics is imagined rather than computed; object placement, motion, and camera trajectories are resolved internally and only visible in the final clip. Inspired by film production, we present Code Video Model, which gives video generation an executable previs, with code as the engine. Given a prompt, Code Video Model writes a program that lays out a scene, animates objects, places a camera, and computes dynamics from equations; rendering the program yields a proxy video and a reference image. Rendering the resulting proxy video takes seconds, making it fast to preview; it is grounded, since dynamics are computed, and directly and precisely controllable, because code can modify each element. A pretrained video model is then conditioned on the proxy video and reference image to generate the final clip. However, we observe that existing video models tend to copy the flat appearance of the proxy video into the generated clip. To address this issue, Code Video Model further introduces early-step conditioning, a training-free schedule that attends to the proxy video only during early sampling steps when structure is established, and drops this conditioning thereafter. As a result, Code Video Model is able to generate a final clip that follows the proxy video’s structure while preserving the reference image’s appearance. We introduce CodeVideoBench to evaluate controllable video generation across diverse applications including gaming, scene editing, world-model data, physics grounding, bullet time, and 3D/4D reconstruction. On CodeVideoBench, Code Video Model with MiniMax-H3 achieves overall scores of 85.2, 82.1, and 85.8 under QwenVL-3.8-27B, GPT-5.6, and GPT-6, respectively, versus 74.3, 73.1, and 80.6 for VACE, the strongest learned baseline. The project page is available at https://anonymous.4open.science/w/CodeVideoModel-2027.

Primary area:
generative models
Keywords:
Video Generation, Controllable Generation, Agentic Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pKMgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Code Video Model. A coding agent turns a prompt into an executable scene program and renders a proxy video, and a frozen video model then generates the final clip. To keep the output from copying the proxy’s flat appearance, the authors propose a training-free strategy that applies proxy conditioning only during the early sampling steps. Experiments on their own benchmark show that it outperforms existing methods.

Critical strengths

This paper uses executable programs as a previs engine for video generation. Scene layout, object motion, camera trajectories, and timing are written explicitly into code, so the structure can be inspected and edited line by line, and the dynamics are computed from equations at the proxy level. The early-step conditioning strategy needs no training and no extra modules, yet it lets a frozen model follow both the proxy’s structure and the reference image’s appearance. The authors analyze the two native conditioning pathways with step-wise sensitivity and feature probes, which explains the mechanism well, and they show that the approach transfers to a closed-source model.

Critical weaknesses

The novelty of early-step conditioning is somewhat limited. Prior work such as SDEdit has already established that structure is mostly set in the early denoising steps, and recent training-free temporal conditioning methods also rely on this pattern. This paper uses a hard switch at a fixed cutoff step, so we suggest exploring content-adaptive schedules and giving a detailed justification.

The case for physical consistency is not yet sufficient. Dynamics are computed only in the proxy video, so after the cutoff the frozen model may drift from the trajectories the equations define, and physical validation relies mainly on qualitative examples. The recent PhysAgent work already uses an execute-and-verify loop, so we suggest quantitatively measuring the physical error in the final videos.

Questions for authors

Please refer to weaknesses.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the careful review and for recognizing the key contributions of our work, including the use of executable programs as a previs engine for video generation, the early-step conditioning strategy, and our analysis on the pathway conditioning.

W1. Part 1. Novelty and Contribution

Our central contribution is using code as an executable previs for video generation, together with early-step conditioning as a technical method for realizing this approach using a frozen R2V model.

Our method is conceptually related to SDEdit only at a high level; technically, it differs substantially in its initialization and conditioning mechanisms.

Table R9: Key methodological differences between SDEdit and Code Video Model.

AspectSDEdit BaselineCode Video Model (Ours)
Initial denoising stateGuide corrupted to an intermediate noise levelRandom noise
How the guide entersThrough the initial denoising stateAs a clean external reference through the model's native encoder and VAE pathways
Control parameterGuide noise level, which determines the starting timestepNumber of initial steps with proxy conditioning (K). The appearance reference remains active throughout.

The following table compares both approaches using the same frozen MiniMax-H3 backbone (the full table appears as Table 1 in our main paper). Our method achieves a higher overall score under all three evaluators.

Table R10: Quantitative comparison between SDEdit and Code Video Model.

MethodQwen-27BGPT-5.6GPT-6
SDEdit64.3768.6669.22
Ours85.1782.0985.82

W1. Part 2. Content-adaptive Schedule for Proxy-Conditioning

We thank the reviewer for suggesting that we explore content-adaptive schedules for proxy conditioning. Figure 4(a) shows that proxy conditioning has a greater influence on generation in early denoising steps than in later ones. Based on this observation, we propose and evaluate an online schedule that selects a cutoff for each sample. At each denoising step ii, we compute the sensitivity of each proxy pathway using Eq. (4) of the main paper and take the maximum of the two.

St,g=∥vtfull−vt∖g∥2∥vtfull∥2,di=max⁡g∈proxy-VAE,,proxy-Qwen-VLSti,g.S_{t,g} = \frac{\left\lVert v_t^{\mathrm{full}} - v_t^{\setminus g}\right\rVert_2}{\left\lVert v_t^{\mathrm{full}}\right\rVert_2}, \qquad d_i = \max_{g \in \\{\text{proxy-VAE},\\,\text{proxy-Qwen-VL}\\}} S_{t_i,g}.

Both velocity predictions use the same latent and timestep, with full conditioning and with pathway gg removed, respectively. At the first update i≥2i \geq 2 satisfying

di−1<τanddi<τ,d_{i-1} < \tau \qquad \text{and} \qquad d_i < \tau,

we disable both proxy pathways from update ii onward, giving K=i−1K=i-1. If the criterion is never met, K=NK=N. Text and reference image conditioning are active throughout. We evaluated three thresholds on CodeVideoBench.

Table R11: Quantitative comparison of fixed and adaptive cutoffs for proxy conditioning.

MethodQwen-27BGPT-5.6 SolGPT-6 Astra
Fixed K = 585.1782.0985.82
Adaptive τ = 0.485.9683.8986.43
Adaptive τ = 0.586.4284.1586.76
Adaptive τ = 0.685.2382.4785.95

The adaptive schedule modestly improved scores under all three evaluators, with τ=0.5\tau=0.5 performing best. Fixed K=5K=5 remains a practical default, offering similar generation quality with fewer model evaluations. The adaptive schedule and its quantitative evaluation are now included in Appendix F of the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

W2. Physical Consistency Evaluation

We thank the reviewer for suggesting a physical consistency evaluation. Following PhysAgent’s open-source evaluation procedure, we provide each judge with the final video, the task prompt, the reference image, and the physical configuration produced by our coding agent during proxy video generation. The judge uses this configuration to assess whether the observed contacts, motion, and object responses are physically plausible and consistent with the intended event.

Qwen-27B, GPT-5.6 Sol, and GPT-6 Astra each assign a score in [0,1][0,1] to every evaluated video. For each method and judge, we average the scores across CodeVideoBench cases and multiply the mean by 100. We report Physical Consistency as a fourth dimension alongside Structure, Style, and Quality.

Table R12: Physical consistency scores of final videos on CodeVideoBench.

MethodQwen-27BGPT-5.6 SolGPT-6 Astra
VACE81.7375.6476.32
Go-with-the-Flow57.2939.4341.38
Diffusion as Shader63.9149.7749.95
Text-to-video83.2377.0574.45
R2V82.4577.5981.77
SDEdit82.5078.3278.73
Code Video Model (Ours)84.0579.1482.73

Our Code Video Model achieves the highest Physical Consistency scores under all three evaluators, consistently outperforming all compared baselines. This demonstrates that our method also achieves strong physical consistency in addition to its performance on the original evaluation dimensions. The evaluation protocol and physical consistency results are now included in Appendix D.2 of the revised manuscript.

Thank you again for your thoughtful review. We would be happy to discuss any remaining questions and would be grateful if you could raise your score.

Dear Reviewer,

The discussion period is nearing its end, and we haven’t yet seen a reply from you. We encourage you to respond to the authors’ comments, especially if you have further questions about points that may affect your assessment.

Thank you for taking the time to engage in the discussion.

Best,

Your AC

Reviewer_HwGKOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a pipeline that combines coding agent and video generation model to generate videos in a "coarse-to-fine" manner. The coding agent plans the scene layout, scene dynamics, and camera movement with "Three.js" code, which generates controllable but coarse structures of the scene and renders proxy video from there. Then, the proxy video is used as a condition for fixed pretrained video models to generate pleasing visual details for the scene. The main technical contribution of the paper is the "early-step" conditioning in the second stage, where it only add the proxy condition at early denoising steps, so that the scene structure relies more on the proxy video at early steps but the visual details rely more on the video gen models at late steps. The paper also conducts VLM-as-judge eval and user preference study to evaluate the proposed pipeline.

Critical strengths

The main strength of the paper is the introduction of the coding agent for video gen. The controllability and physical grounding capability of coding agent are exactly where the video models fall short in. Whereas pre-trained video generation models can provide much better visual details for the generated proxy videos from coding agent. The combination of these two exactly leverages their strengths. To achieve that, the paper proposes "early-step" conditioning for the video generation stage and analyzed the effects of conditioning steps for the final generated videos. As coding agents and video generation models become more and more powerful, the combination of these two would produce even stronger results.

Critical weaknesses

Weakness 1: The major weakness if that even though the "early-step" conditioning improves the appearance, it still changes the object layout and scene structures. Even with the best choice of step, there is still some noticeable misalignments of scene structure, layout, and object details between proxy videos and generated videos. As an example, in Fig. 2 of the main paper, the elevator doors position and orientation was changed, the desktop objects and text were changed, the person's sitting position and direction, as well as the scene's camera position were changed.

Furthermore, in the user preference study, in "Structure" column, which is one of the main claimed contribution, the proposed method only ranks three among baselines, which is lower than R2V and SDEdit.

Weakness 2: Another weakness is that the generated coding assets from the coding agent seems error-prone and may fail to generate detailed structures when provided with a detailed prompt. The subsequent video generation stage would likely introduce more artifacts on the details, such as the desktop objects in Fig. 2.

Questions for authors
  1. Did the authors provide an analysis on the prompts used for generating scenes? In the generated demo video, it seems some prompts are quite short with no detailed prompts such as the object structures etc. It is suggested to analyze the prompt distribution and identify the types of prompts for which the pipeline performs well or poorly at.
  2. Did the authors perform any prompt engineering or harness for coding agent to guide it to generate better proxy videos in the first place? And how would different qualities of proxy videos influence the final generated videos?
  3. For the videos involved with physical dynamics, how were the prompts defined?
  4. Are the coding prompts iteratively refined over multiple rounds to achieve the desired layout, camera trajectory, and physical dynamics, or are they defined only once?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your thoughtful review and for recognizing the key contributions of our work, including the use of coding as a video generation engine, the underlying motivation, and the integration of agent-based reasoning with diffusion models to leverage their complementary strengths.

W1. Part 1. Misalignments

We thank the reviewer for pointing out the misalignments in Fig. 2 that we previously overlooked. These discrepancies mainly stem from inconsistencies between the reference image and the proxy video. To address this, we generated new results using reference images better aligned with the proxy videos. Better input alignment can improve structural consistency in the generated video. Updated results and a reference–proxy alignment comparison are provided in Figure 2(a) and Figure 2(b), respectively, with further discussion in Section 4.3 of the revised manuscript.

W1. Part 2. Structure Following

Our method aims to preserve structure while adapting textures and appearance to the reference image, and there is a trade-off between these objectives. In practice, R2V and SDEdit achieve higher Structure scores because their outputs remain overly close to the proxy video, making minimal changes to texture and appearance. Thus, an excessively high Structure score here actually reflects a failure mode — over-preserving the proxy at the expense of appearance control.

W2/Q1. Effect of Prompt Lengths on Video Generation

For each coding model, we generated proxy videos for the same set of scenes using short (<50 words), standard (50–150 words), and long (>150 words) prompts, averaging 43.41, 94.68, and 178.77 words, respectively. GPT-6 Astra evaluated each proxy video on two criteria, assigning a separate score on a 0-100 scale for each. Semantic Consistency measures how well the video matches the intended scene, subjects, main actions, and sequence of events. Detail Alignment measures how well the video captures explicitly requested details, including appearance, object components, spatial relationships, and occlusion. Each cell reports Semantic Consistency / Detail Alignment.

Table R5: Proxy video quality across coding models and prompt lengths, measured by semantic consistency and detail alignment.

Coding ModelShort (<50 words)Standard (50-150 words)Long (>150 words)
GPT-5.6 Luna76.50 / 65.1289.62 / 81.7590.38 / 82.62
GPT-5.6 Sol94.25 / 84.6295.88 / 89.8896.62 / 91.50
GPT-6 Astra97.72 / 96.1297.75 / 95.5097.98 / 96.25

The results suggest that longer prompts enable less capable LLMs to generate higher-quality proxy videos (see the GPT-5.6 Luna row). In contrast, more capable LLMs (e.g., GPT-6 Astra) are less sensitive to prompt length, achieving comparable proxy quality across different prompt lengths. This analysis is now included in Appendix E.1 of the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

W2. Part 2. Artifacts in the Subsequent Video Generation Stage

We used the proxy videos evaluated in W2/Q1 to condition downstream video generation. The table below reports the overall scores of the resulting videos for each coding model and prompt length. Each cell lists the scores assigned by Qwen-27B, GPT-5.6 Sol, and GPT-6 Astra, in that order.

Table R6: Final video quality across coding models and prompt lengths using the corresponding proxies from Table R5.

Coding ModelShort (<50 words)Standard (50-150 words)Long (>150 words)
GPT-5.6 Luna72.53 / 74.31 / 77.7477.85 / 77.98 / 81.0178.56 / 79.24 / 82.23
GPT-5.6 Sol83.14 / 79.93 / 83.1283.51 / 80.76 / 83.3984.16 / 81.87 / 84.66
GPT-6 Astra84.79 / 81.86 / 85.3785.20 / 82.12 / 85.9485.37 / 82.59 / 86.23

Final video scores increased with prompt length for all three coding models under every evaluator. GPT-5.6 Luna showed the largest gains in final video scores, consistent with its gains in proxy quality in W2/Q1. The results suggest that improvements in proxy quality can carry through to the final video. This analysis is now included in Appendix E.1 of the revised manuscript.

Q2. Effect of Coding Agent Harness on Video Generation

We thank the reviewer for suggesting the study of the harness effect. Our pipeline uses Copilot as the default coding agent harness. To assess the impact of harness choice, we also evaluated Claude Code, with its more elaborate workflow, and Pi, with its lightweight, extensible design. We used the same CodeVideoBench prompts and the same underlying model, GPT-5.6 Sol with high reasoning effort, across all three harnesses.

Table R7: Semantic consistency and detail alignment by coding agent harness.

HarnessSemantic ConsistencyDetail Alignment
Claude Code92.2587.25
Copilot94.8889.88
Pi95.3890.38

The results show that Pi scored highest on both criteria, closely followed by Copilot, with Claude Code ranking third. This suggests that harness choice can affect proxy video quality, while a more elaborate workflow does not necessarily yield better results. This analysis is now included in Appendix E.2 of the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

Q3. Prompt Design for Physical Dynamics

We use scene-specific prompts together with a shared set of physics-oriented coding instructions. Each scene-specific prompt describes the physical setup, initial conditions, triggering action, and intended causal sequence. The shared instructions require each scene to be implemented in Three.js and coupled with the Rapier rigid-body physics engine. All principal moving objects must be represented as simulated rigid bodies with appropriate colliders and, where necessary, joints. The simulation uses standard gravity and permits initial velocities or impulses, physically motivated forces, and explicitly defined kinematic actuators.

Q4. Iterative Program Generation and Thinking-Level Analysis

Our Three.js program is generated through multiple iterations. Specifically, we use Copilot's /goal mode, which enables the coding agent to autonomously perform multiple render-inspect-revise iterations within a single task session.

The agent iteratively renders and inspects keyframes and contact sheets, then revises the Three.js program to improve spatial layout, camera trajectory, and object motion. The visual feedback from each iteration is incorporated into the agent's context to guide subsequent revisions, while the original task specification remains unchanged. Once the iterative refinement is complete, the finalized program is rendered into a fixed proxy video for downstream video generation.

Furthermore, we investigate the effect of the coding model's thinking level on proxy video quality under the same Copilot /goal workflow. We compare three thinking levels (Low, High, and Max) across three coding models. The resulting proxy videos are evaluated using the two criteria defined in W2/Q1, with each entry reporting Semantic Consistency / Detail Alignment.

Table R8: Semantic consistency and detail alignment by coding model and reasoning effort.

Coding ModelLowHighMax
GPT-5.6 Luna72.88 / 60.1289.62 / 81.7592.50 / 85.25
GPT-5.6 Sol92.00 / 86.2595.88 / 89.8896.88 / 93.88
GPT-6 Astra97.25 / 94.3897.75 / 95.5097.88 / 96.38

The results suggest that increasing thinking effort leads to better proxy video quality for relatively less capable models. In contrast, highly capable models are less sensitive to the choice of thinking effort, with only marginal differences in proxy quality across different effort levels. The iterative refinement procedure and reasoning-effort analysis are now described in Appendix E.3 of the revised manuscript.

Thank you again for your thoughtful review. We would be happy to discuss any remaining questions and would be grateful if you could raise your score.

Dear Reviewer,

The discussion period is nearing its end, and we haven’t yet seen a reply from you. We encourage you to respond to the authors’ comments, especially if you have further questions about points that may affect your assessment.

Thank you for taking the time to engage in the discussion.

Best,

Your AC

Reviewer_2WsmOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Code Video Model, a training-free framework that uses a coding agent to generate executable Three.js programs and control video generation. The rendered proxy video and an appearance reference guide a frozen video model, with proxy conditioning restricted to early denoising steps to reduce synthetic appearance leakage. Experiments on the 22-case CodeVideoBench, including baseline comparisons, ablations, and human evaluation, show improved overall results.

Critical strengths

This paper proposes a practical framework that leverages existing coding agents and video generation models to achieve controllable video generation. The provided videos and comparisons show competitive results.

Critical weaknesses
  1. The eval set is limited. CodeVideoBench comprises only 22 cases; for five of the eight application categories, there are merely two cases each. Relying solely on overall mean values ​​is insufficient to support conclusions regarding robustness across different applications.

  2. The main technical novelty is early conditioning. The method directly integrates code generation and pretrained video models, injecting proxy conditions only during the first K denoising steps (25% by default). However, K appears to be heuristic. Does it generalize across samples, or should it be smaller when the code-based modeling is inaccurate?

Questions for authors
  1. Additional evaluation: Could you evaluate the method on a larger set, such as several hundred videos, to assess whether the reported improvements hold across applications?

  2. Does the default value of K generalize across samples, or should it be reduced when the code-based modeling is inaccurate?

9 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the careful review and for recognizing that Code Video Model is a practical framework with compelling results.

W1/Q1. Additional Evaluation

We expanded CodeVideoBench from 22 to 220 evaluation cases, increasing the number of cases in each of the eight application categories by a factor of ten. This expanded benchmark, namely CodeVideoBench-Plus, will be released. Table R1 reports the overall and per-application scores of Code Video Model and the baselines on CodeVideoBench-Plus, with methods in rows and the average scores in the first score column. Each entry lists the scores assigned by Qwen-27B / GPT-5.6 Sol / GPT-6 Astra, in that order. Bold scores denote the best-performing methods, respectively. We highlight that Code Video Model ranks first in almost all per-application comparisons and achieves the highest average score under all three evaluators. The expanded benchmark is described in Section 4.1 of the revised manuscript, with results presented in Figure 10 and discussed in Section 4.2. An anonymous browser of the 220 CodeVideoBench-Plus cases, including proxy videos, reference images, and prompts, is available here.

Table R1: Quantitative comparison with baselines across eight applications on CodeVideoBench-Plus.

MethodAverageGamingBullet TimePhysical GroundingScene / World EditingProduct CinematographyArchitectural CinematicsTrajectory VariationRobotics Simulation
VACE73.27 / 73.20 / 79.5675.14 / 65.09 / 70.5650.79 / 54.78 / 58.7975.56 / 86.36 / 84.6980.04 / 84.49 / 84.9354.54 / 69.35 / 74.7981.44 / 83.92 / 82.6579.77 / 65.25 / 79.0174.44 / 73.11 / 87.90
Go-with-the-Flow47.48 / 54.27 / 64.7867.38 / 56.16 / 67.8641.62 / 57.69 / 49.8756.92 / 58.42 / 57.3630.09 / 42.44 / 58.6444.27 / 41.56 / 50.3936.21 / 47.06 / 55.5446.40 / 55.88 / 73.5661.45 / 69.72 / 83.58
Diffusion as Shader58.29 / 66.61 / 71.9659.68 / 59.12 / 67.7562.93 / 66.91 / 67.5464.14 / 62.16 / 63.2545.63 / 57.30 / 62.7944.13 / 54.43 / 68.1458.15 / 62.43 / 64.7660.41 / 79.05 / 82.6770.04 / 77.48 / 84.60
Text-to-video62.76 / 66.34 / 69.1661.32 / 56.66 / 62.6076.98 / 73.22 / 76.7652.73 / 72.33 / 68.1180.74 / 63.75 / 68.8667.57 / 68.06 / 69.4958.76 / 68.06 / 70.4753.85 / 64.76 / 68.2551.92 / 67.18 / 69.58
R2V74.19 / 77.75 / 79.9365.24 / 75.13 / 79.4363.94 / 71.63 / 79.0974.67 / 75.43 / 72.2066.65 / 73.82 / 72.5567.88 / 81.95 / 75.6492.88 / 84.73 / 82.5584.26 / 82.04 / 90.3474.83 / 77.33 / 82.27
SDEdit63.42 / 67.59 / 68.3364.24 / 58.22 / 65.3257.58 / 56.78 / 63.8566.88 / 72.48 / 68.3864.34 / 69.12 / 69.8862.03 / 70.39 / 63.6564.89 / 61.10 / 67.4360.50 / 70.40 / 70.0366.16 / 72.74 / 71.59
Code Video Model (Ours)85.29 / 81.59 / 85.2174.09 / 75.66 / 79.8285.03 / 80.71 / 81.0485.65 / 86.85 / 87.5681.99 / 84.66 / 85.0880.73 / 74.72 / 77.9292.53 / 85.05 / 83.3186.34 / 84.14 / 90.6591.75 / 78.45 / 88.10

The CodeVideoBench-Plus results across all eight applications are included in the updated Figure 10(b) of the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

Table R2 reports the overall and per-application scores of Code Video Model and the baselines on CodeVideoBench, with methods in rows and the average scores in the first score column. Each entry lists the scores assigned by Qwen-27B, GPT-5.6 Sol, and GPT-6 Astra, in that order.

Table R2: Quantitative comparison with baselines across eight applications on CodeVideoBench.

MethodAverageGamingBullet TimePhysical GroundingScene / World EditingProduct CinematographyArchitectural CinematicsTrajectory VariationRobotics Simulation
VACE74.29 / 73.10 / 80.5977.65 / 65.39 / 74.6855.32 / 57.14 / 61.3574.55 / 83.86 / 82.5082.74 / 86.21 / 86.9058.93 / 71.92 / 77.7785.50 / 86.51 / 87.4377.27 / 62.75 / 78.4172.61 / 70.68 / 86.07
Go-with-the-Flow48.48 / 55.23 / 65.8470.35 / 58.19 / 70.1242.60 / 57.99 / 52.6554.42 / 56.60 / 55.4332.74 / 42.74 / 60.4647.75 / 45.55 / 54.3239.10 / 51.10 / 56.0847.84 / 59.09 / 76.2258.95 / 67.22 / 81.14
Diffusion as Shader59.28 / 67.66 / 72.9064.49 / 63.04 / 71.6065.14 / 69.69 / 71.9363.22 / 60.91 / 61.6349.29 / 62.30 / 67.7942.07 / 53.08 / 65.9361.76 / 64.90 / 67.7159.80 / 77.90 / 81.0268.61 / 76.12 / 82.74
Text-to-video63.80 / 67.59 / 69.0964.07 / 57.61 / 64.3272.48 / 70.72 / 74.2652.26 / 66.81 / 65.6082.95 / 68.93 / 69.2267.89 / 71.29 / 69.7362.96 / 71.56 / 70.7855.99 / 65.06 / 68.5952.13 / 68.76 / 69.84
R2V75.22 / 78.72 / 80.9466.41 / 81.95 / 80.8967.43 / 79.04 / 81.0077.12 / 81.02 / 76.5569.13 / 76.16 / 75.3571.40 / 75.94 / 79.4490.40 / 86.51 / 85.6485.02 / 79.62 / 88.2973.18 / 74.95 / 79.77
SDEdit64.37 / 68.66 / 69.2262.59 / 57.75 / 63.7362.11 / 61.77 / 66.5964.77 / 71.88 / 67.3869.34 / 73.93 / 73.6765.94 / 73.43 / 68.1062.80 / 59.23 / 66.2459.91 / 69.28 / 69.7565.68 / 72.39 / 71.27
Code Video Model (Ours)85.17 / 82.09 / 85.8276.16 / 80.10 / 81.7789.53 / 84.40 / 85.4982.76 / 84.74 / 86.5782.85 / 86.35 / 87.0082.84 / 76.50 / 79.5494.30 / 86.61 / 87.5384.04 / 82.74 / 88.3988.75 / 76.23 / 86.17

The CodeVideoBench results across all eight applications are included in the updated Figure 10(a) of the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

W2/Q2. Analysis of Proxy-Conditioning Duration

Please kindly refer to Table 2 and Figure 4(a) in the main paper for the ablation study. The choice of K=5K=5 is supported by the sensitivity analysis in Figure 4 and its superior performance in the ablation study.

Table 2 of the main paper evaluates different proxy-conditioning durations and schedules on CodeVideoBench. Conditioning during the first five denoising steps (K=5K=5) achieves the highest mean score under all three evaluators, outperforming conditioning during the first three steps, the first seven steps, and the last five steps.

Table R3: Ablation of the timing and duration of proxy video conditioning.

Proxy video conditioning (N = 20)Qwen-27BGPT-5.6GPT-6
First 3 steps81.8379.9083.76
First 5 steps (ours)85.1782.0985.82
First 7 steps82.9180.5784.98
Last 5 steps69.4371.7974.25

The step-wise sensitivity analysis presented in Figure 4(a) of the main paper further supports this finding, showing that proxy conditioning has its strongest influence during the early denoising steps. Together, these results support K = 5 as a robust default across the evaluated test cases.

W2. Part2. Robustness to Proxy Quality

To evaluate whether lower-quality proxies benefit from shorter conditioning, we used GPT-5.6 Luna, GPT-5.6 Sol, and GPT-6 Astra to generate proxy videos of varying quality. For the proxies from each model, we applied conditioning during the first KK denoising steps, with K∈{3,5,7}K\in\{3,5,7\}. Each cell reports the overall scores assigned to the final videos by Qwen-27B, GPT-5.6 Sol, and GPT-6 Astra, in that order.

Table R4: Conditioning-duration ablation across coding models.

Coding ModelK = 3K = 5K = 7
GPT-5.6 Luna76.07 / 76.53 / 79.9777.93 / 77.94 / 81.1277.98 / 76.99 / 80.51
GPT-5.6 Sol80.95 / 78.42 / 81.8383.47 / 80.67 / 83.4582.04 / 79.63 / 83.01
GPT-6 Astra81.83 / 79.90 / 83.7685.17 / 82.09 / 85.8282.91 / 80.57 / 84.98

K=5K=5 achieved the highest scores in the vast majority of comparisons across coding models and evaluators, supporting its use as a robust default. This analysis is now included in Section 4.3 of the revised manuscript, with quantitative results reported in Table 4.

We thank the reviewer for the thoughtful review. We would be happy to discuss any remaining questions and would be grateful if you could consider raising your score.

Comments on the paper

3 comments

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. We are encouraged by their recognition of the practical value of code-driven video generation, the executable previs representation, and the early-step conditioning strategy. In response, we have expanded our evaluation, conducted additional ablations, and clarified the method and implementation details. The main updates are summarized below.

Expanded Benchmark and Robustness of Proxy Conditioning

In response to Reviewer 2Wsm, we expanded CodeVideoBench from 22 to 220 cases to construct CodeVideoBench-Plus, increasing the number of cases in each application category tenfold. Sections 4.1–4.2 and Figure 10 of the revised manuscript present the expanded benchmark and per-application comparisons. Code Video Model achieves the highest average score under all three evaluators on both benchmarks. We also evaluated conditioning durations across proxies generated by different coding models. The results in Section 4.3 and Table 4 show that K=5K=5 performs best in the vast majority of comparisons, supporting its use as a robust default. An anonymous CodeVideoBench-Plus page provides the 220 cases with proxy videos, reference images, and prompts.

Reference Alignment and Coding-Agent Configuration

To address Reviewer HwGK’s concerns, we revisited reference–proxy misalignment and generated updated results using better-aligned references. Figure 2(a) presents the updated results, while Figure 2(b) compares aligned and misaligned references, showing improved structural consistency and fewer visual artifacts with better alignment. Section 4.3 discusses these observations alongside visual conditioning interactions. We further evaluated the effects of prompt length, coding-agent harness, and reasoning effort on proxy quality, together with the downstream effects on final-video quality. These experiments and a description of the iterative program-refinement workflow are provided in Appendix E.

Adaptive Conditioning and Physical Consistency

Following Reviewer pKMg’s suggestion to explore content-adaptive conditioning, we developed and evaluated a sensitivity-based schedule that selects a cutoff for each sample. Appendix F reports modest improvements over fixed K=5K=5, while the fixed schedule remains a practical trade-off between generation quality and inference cost. We also added a Physical Consistency evaluation following PhysAgent’s open-source procedure. The protocol and results in Appendix D.2 show that Code Video Model achieves the highest scores under all three evaluators. In our responses, we additionally clarify the methodological differences from SDEdit, particularly the denoising initialization and the use of a clean external proxy reference.

We have reorganized the supplementary material and added cross-references in the main text to help reviewers locate the new analyses. New and substantively revised text, together with the relevant figure and table captions, is marked in blue in the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer HwGK,

Thank you for following up on the reference images in Fig. 2 and the high proxy scores in Table R5. We address the concerns as follows.

1. How we obtain the reference image and when its layout matters (W1)

how these better-aligned reference images were obtained?

In our reference-generation procedure, we take a representative frame from the rendered Three.js sequence and pass it to GPT-Image-2. We ask the editor to preserve the camera viewpoint, object positions, and poses while changing the materials, textures, and lighting. For the updated results in Fig. 2(a), the coding agent additionally checks the edited reference against the first proxy frame for alignment before video generation. By “better aligned” we mean closer agreement in the spatial cues visible in both inputs, such as where the elevator doors appear and which way the seated person faces. Fig. 2(b) compares generated results using misaligned and aligned references for the same proxy videos.

Were the reference images selected based on the generated proxy videos?

Yes, the proxy guides how we construct the appearance reference.

Does preserving layout and fine details depend on how closely the reference image matches the proxy?

Yes, reference–proxy layout conflicts can affect structural fidelity. The proxy-guided appearance-editing workflow and coding-agent alignment check help obtain compatible inputs, as illustrated in Fig. 2.

AuthorsOpenReview ↗

Official Comment by Authors

2. W2/Q1

How reliable and consistent is this evaluation methodology?

Table R6 shows consistent rankings across evaluators. For each prompt length, Qwen-27B, GPT-5.6 Sol, and GPT-6 Astra ranked the final videos generated from the three coding models' proxies with Astra first, Sol second, and Luna third. All three also found that Luna benefited most from longer prompts. This agreement corroborates the downstream rankings and prompt-length trends across evaluators.

Could GPT-6 Astra favor its own outputs?

The ranking in Table R6 holds under both other judges. At every prompt length, Qwen-27B and GPT-5.6 Sol agree with Astra on the ordering of the final videos. For standard-length prompts, Qwen scores videos made from Astra, Sol, and Luna proxies 85.20, 83.51, and 77.85, respectively. Thus the observed trend holds when Astra is excluded as an evaluator. Whether Astra assigned systematically higher absolute scores to its own proxies in Table R5 remains an open question, since only Astra scored those proxies directly.

Each cell in Table R6 gives mean final-video scores for short / standard / long prompts.

Table R6. Mean final-video scores by coding model, prompt length, and evaluator.

Coding modelQwen-27BGPT-5.6 SolGPT-6 Astra
GPT-5.6 Luna72.53 / 77.85 / 78.5674.31 / 77.98 / 79.2477.74 / 81.01 / 82.23
GPT-5.6 Sol83.14 / 83.51 / 84.1679.93 / 80.76 / 81.8783.12 / 83.39 / 84.66
GPT-6 Astra84.79 / 85.20 / 85.3781.86 / 82.12 / 82.5985.37 / 85.94 / 86.23

3. Relative alignment of spatial relationships, layout, and fine details

How reliable is this evaluation approach, particularly for assessing fine-grained details such as spatial relationships, layout, and geometric structures?

We use these scores for relative comparisons between methods under the same prompts and rubric. The absolute values are aggregate judgments on the evaluator's scoring scale. Even a score of 95–97 can coexist with local errors in object placement, orientation, or occlusion.

The qualitative comparison in Fig. 9 is consistent with our higher overall scores in Table 1, showing a better balance between following the proxy structure and matching the reference appearance in the illustrated cases.

Thank you again for the valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.