acceptodds
Withdrawn from ICLR 2027

ProxyUp: Training-Free Generation of Coherent Video Interactions from Motion Proxies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Motion proxies can specify an action without depicting the objects, contacts, and scene needed to realize it naturally. We study how to turn such incomplete guidance into visually natural videos with coherent interactions while retaining the intended actions and key event structure. We propose ProxyUp, a training-free framework built on a pretrained video generator. Its inference design combines region-wise proxy initialization with full-latent refinement using predict-and-perturb updates, followed by ODE sampling. The mask selects initial guidance but does not restrict subsequent updates: the guided object and new content can adapt together at contacts and occlusions. The proxy thus guides generation without becoming a framewise target. On simulation and real-world proxies, ProxyUp achieves higher event-based motion and mechanics scores, imaging quality, and mean human ratings than the evaluated baselines.

Primary area:
generative models
Keywords:
video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hdkMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ProxyUp, a training-free video generation method that uses motion proxy videos as input to generate realistic scenes with coherent interactions. The method mainly combines two existing techniques: (1) region-wise latent noising (RLN), which applies SDEdit-style latent initialization to the masked foreground of a source video while replacing the background with noise, and (2) stochastic flow relaxation (SFR), which directly adopts the Predict-and-Perturb (P&P) strategy from Self-Refining Video Sampling. The resulting latent is then denoised using a pretrained Wan2.2 model. Experiments show improvements over several video editing and motion transfer baselines.

Critical strengths
  1. The idea of using incomplete motion proxies to generate realistic video interactions is interesting.
  2. The qualitative results are generally promising, especially for examples involving object interactions.
Critical weaknesses
  1. Limited technical novelty. This is my main concern. The proposed framework seems to be a straightforward combination of existing video editing and sampling techniques, with the main difference being the use of motion proxies as source videos. RLN is essentially masked inversion and noise initialization, while SFR directly uses the existing P&P strategy. I do not see a substantial technical contribution beyond adapting these methods to a different type of input video.
  2. Insufficient experimental comparisons. The paper compares against several existing video editing methods, but does not include some natural baselines, such as SDEdit combined with P&P refinement. Such comparisons are important because the proposed pipeline is largely built from these existing components. The current ablations only show that RLN and SFR are useful, but do not demonstrate that their combination is better than simpler alternatives under the same computational budget.
  3. Unclear practical applicability. The method assumes that a suitable motion proxy is already available. The evaluation uses simulated videos and existing real-world recordings, but it is unclear how users would obtain these proxies in practical applications. In some cases, creating a proxy with the desired motion may itself require considerable effort.
  4. High inference cost. Despite being training-free, the method requires 32 inversion steps, 15 refinement iterations, and 32 sampling steps. This is considerably more expensive than standard video generation in terms of sampling operations. However, the paper does not report inference time or computational overhead, making it difficult to assess the practicality of the approach.
  5. Limited evaluation of motion preservation. The primary motion-related metric (MR) evaluates whether the generated video completes the events described in the prompt, rather than measuring how closely its motion follows the input proxy. Although the paper reports DPS in the appendix, it does not compare this metric against the baselines. As a result, it remains unclear how effectively the method actually preserves proxy motion.
Questions for authors
  1. What is the key methodological contribution beyond applying masked latent initialization and existing P&P refinement to motion proxy videos?
  2. Could the authors compare against SDEdit with P&P refinement and other straightforward combinations of existing techniques under matched inference budgets?
  3. How are motion proxies expected to be obtained in practical applications, especially when no suitable simulation or reference video is available?
  4. What is the actual inference time of ProxyUp compared with Wan2.2 and the other baselines?
  5. Could the authors provide direct quantitative comparisons of motion preservation using DPS or trajectory-based metrics?
Reviewer_BghqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces proxyup, a training free conditioned video generation framework from incomplete motion proxies. The authors use simplified and maksed videos that represent the desired motions and interactions of the objects in the scene and guide the generation of the video. Region wise latent noising initializes the foreground from an inverted proxy and the background from fresh noise. Full latent refinement then applies an existing predict and perturb update, allow a flexible generation of the guided objects before the final denoising.

Critical strengths
  1. This method give a more explicit and straightforward way of motion control in video generation.
  2. It is a training free pipeline.
  3. Selective initialization and unrestircted joint refinement gives more freedom to video gen to complete the scene.
Critical weaknesses
  1. The pipeline is actually too costly and inpractical for production. It is very hard to get those proxy video, though blenders or mpm and etc. Also the proxies themselves are not garuantted to be accurate, it is very hard to adjust those physical parameters to get good physical references.
  2. The reliability of the gains is hard to evaluate. The mechanics and material scores has limited evidence in physical coherence.
  3. The core refinement update are inherited from self-refining video sampling, and the contribution of refinement gain is not well established.
Questions for authors
  1. On Wan2.2 with equal forward pass budgets and identical prompts, how does proxyup compare with text only P&P, full video initialization plus P&P, and RLN with the original P&P schedule?
  2. How much manual effort and simulation tuning were required to construct the evaluated proxies, especially the MPM cutting and fluid examples?
Reviewer_LRGbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ProxyUp addresses training-free video generation from incomplete motion proxies, aiming to preserve recognizable action structure while synthesizing missing interaction partners, contacts, and scene content. The method combines region-wise latent noising, which initializes selected regions through proxy inversion and replaces the background with noise, with full-latent Predict-and-Perturb refinement and subsequent ODE sampling.

Critical strengths

A useful problem formulation with a sensible design. Generating complete interactions from incomplete proxies requires allowing the guided object itself to change, particularly when new hands, tools, or occlusions are introduced.

Critical weaknesses
  1. The evaluation does not sufficiently isolate the contribution of temporal proxy information. Target prompts are written from the proxies and already specify important motion events, while automatic MR evaluates prompt-defined criteria without directly comparing generated and proxy videos. Consequently, higher MR cannot establish whether the method follows temporal information supplied by the proxy or reconstructs the action mainly from text and pretrained priors.
  2. The benefit specific to the proposed combination remains insufficiently established. SFR adopts the basic update from Self-Refining Video Sampling, and the main comparisons differ in backbone and inference budget. Component ablations help, but adding RLN increases MR by only 0.03 on real proxies and 0.02 on simulated proxies, without uncertainty estimates establishing the reliability of these increments.
Questions for authors

Under a fixed backbone and matched model-evaluation budget, can you demonstrate that changing the proxy’s event order or timing changes the generated action accordingly while keeping the same compatible, underspecified prompt, and compare this behavior against text-only P&P and static-proxy controls?

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your time and thoughtful reviews. Please be reminded that the author-reviewer discussion period ends on November 18.

If you haven’t already, please review the authors’ rebuttals and actively participate in the discussion. If you have any remaining questions or concerns, please raise them before the deadline so the authors have sufficient time to respond.

Thank you again for your valuable contributions to ICLR 2027.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.