acceptodds
Under review as a conference paper at ICLR 2027

Dancing Diffusion: Learning Continuous Programs for Image Generation and Editing

Abstract

Diffusion models generate samples through a multi-step denoising trajectory, yet condition every step on the same instruction, so the trajectory cannot be addressed at the level of individual operations. We introduce Dancing Diffusion, which assigns each span of timesteps a dedicated continuous condition block and thereby turns conditioning into a latent program. A compiler maps text features into this program, and a fixed pretrained generator executes its blocks over successive sampling intervals. The trajectory can then dance, keeping its executed history while its continuation is redirected. A shared prefix is executed once and reused, and a later operation is rewritten while content fixed by earlier operations is preserved. The compiler is trained through the images it produces, with visual feedback on both the requested change and the retained content. On SANA, reusing an executed prefix preserves markedly more of the scene than regenerating from noise at a similar strength of edit, learned conditions preserve more background than native prompt switching from the same saved state, and the learned program improves generation quality over native SANA. Later branches weaken the edit, which leaves semantic control over individual operations limited. By converting a single-instruction trajectory into a program of addressable operations, Dancing Diffusion lays a foundation for structured intermediate computation in diffusion models.

Primary area:
generative models
Keywords:
diffusion models, text-to-image generation, image editing, temporal conditioning, latent programs, controllable generation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_tZuuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Dancing Diffusion, a novel framework that transforms the traditional single-instruction diffusion sampling trajectory into a program of addressable operations. While standard diffusion models condition every step on the same static prompt, Dancing Diffusion assigns dedicated continuous condition blocks to different time intervals using a learned compiler. This enables the generation path to "dance"—meaning the model can execute and preserve an early shared prefix history while dynamically redirecting its subsequent continuation toward alternative target instructions.

Critical strengths

Traditional diffusion models condition every sampling step on the same static instruction, preventing intervention or addressing at the level of individual operations. The paper's compiler-executor architecture maps text features into dedicated continuous condition blocks across temporal intervals, effectively turning conditioning into a latent program. This innovation allows the generation trajectory to "dance"—preserving executed history while redirecting its future continuation.

By resuming a saved state and reusing an executed prefix, the method significantly enhances background and scene structure retention compared to full regeneration (e.g., reducing background LPIPS by up to 36% at early branches) and outperforms native prompt switching. Crucially, it achieves this improved retention while maintaining or enhancing image quality, lowering the FID below native SANA, and saving 22% to 23% in computation time when producing multiple continuations from a cached prefix.

Critical weaknesses

The authors acknowledge that semantic control over individual operations remains limited because later branches weaken the edit. Furthermore, block interventions demonstrate routing rather than a true training-induced semantic factorization of the conditioning blocks. This reveals a gap between the conceptual motivation of the method and its actual operational mechanism.

Cross-schedule comparisons combine schedule variations with disparate initialization histories (such as joint warm starts versus training from scratch), which limits causal interpretations across schedules. Additionally, the automated multimodal judge evaluations show no consistent trained-compiler advantage alongside substantial order disagreement, which weakens the empirical robustness of the editing efficacy claims.

Questions for authors

Given that block interventions primarily demonstrate routing rather than a true training-induced semantic factorization, can the authors clarify how to achieve more precise semantic control over individual operations, especially since later branches weaken the edit effect?

Because cross-schedule comparisons mix schedule choices with disparate initialization histories (e.g., 3/9/18 starting from scratch while 6/12/12 and 9/12/9 inherit a 40-update joint warm start), how can we isolate the true causal impact of the schedule from these initialization differences?

Reviewer_4RJyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Dancing Diffusion, an editable conditioning interface for image generation and editing. It introduces a compiler that maps text features of the prompt into a set of conditions for each block of successive diffusion denoising steps. These conditions are supplied to a frozen generator. The partition of the total number of steps into blocks is a hyperparameter. The compiler is trained with a special objective designed to retain some content while completing the requested change. For image editing, the learned suffix at an early branch preserves background better than a native suffix from the same saved state. There is a small FID gain on two of three schedules compared to native SANA.

Critical strengths

The experimental design is careful and answers several questions. It compares learned suffix to full regeneration, native suffix on identical saved state, FlowEdit–SANA, and SDEdit–SANA. Prefix caching cost analysis, same-prompt drift, and single-block interventions complete these comparisons and characterize the mechanism.

Critical weaknesses
  1. The blocks are not shown to cleanly separate their intended semantic roles. In Appendix M, background-conditioned substitutions in B0/B1 also alter the subject's appearance or pose. B0 readout can already contain subject information. In Appendix N, even when B0 is made subject-independent using background-only input, subject changes made through B1/B2 still change the background. The same temporal pattern is already present at initialization, so edit-only training is not shown to create that separation or improve it. Role-mask fitting is the stage that would have to assign the roles, but it is not described in detail, and nothing shows that it was necessary or that it succeeded.
  2. There is little evidence of the benefit of the further training of the compiler. Pairwise Qwen judge does not demonstrate clear trained-compiler advantage over native suffix or its initial state (Tables 12-14). In learned vs. native suffix comparison, both sides still resume a compiler state, so this comparison does not measure the compiler’s contribution over native SANA. The gain itself is not conclusive: it preserves background better (at an early branch), while replacement counts do not consistently favor the learned suffix. FID relative gains are too small.
Questions for authors
  1. Why multiple conditioning blocks? Can you compare your schedules against using just one conditioning block for the whole trajectory, under the same training and the same branch points?
  2. The native suffix resumes a state from the role-fitted compiler, then applies native caption features. Could you compare overall pipelines: the learned suffix vs. prompt switching in native SANA, with no compiler on the prefix or the suffix? Additionally, role-mask fitting is not described in detail. What is its fitting objective? Could you run the same post-training from a compiler that was not role-fitted, to identify the incremental benefit of the role-fitting stage?
Reviewer_YR5vOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Dancing Diffusion, which maps text prompts into timestep-specific continuous conditioning blocks for a frozen diffusion model. By reusing intermediate sampling states and changing subsequent conditions, it enables alternative generation paths. The compiler is trained with feedback on semantic changes and content preservation.

Critical strengths

The compiler–executor framework provides a structured interface for controlling different stages of generation without modifying the pretrained generator.

Critical weaknesses

The method requires saved intermediate states and cannot directly edit arbitrary images. Existing approaches such as FlowEdit already offer competitive editing and preservation without additional training, making the need for a large learned compiler unclear.

The experiments mainly demonstrate temporal conditioning and trajectory reuse, both achievable with existing methods. Learned conditions provide relatively modest gains over native prompt switching, while block interventions do not establish meaningful semantic specialization.

Questions for authors

Why does the evaluation focus only on SANA, rather than stronger models such as FLUX or Qwen-Image?

The visual results appear less appealing than native generation and seem less competitive with FlowEdit. Is this an inherent limitation of the method, or are these simply unfavorable examples?

I am somewhat confused about the positioning of the task. FlowEdit turns a pretrained generation model into an image editor in a training-free manner, while the proposed method requires additional training and cannot directly operate on existing images. Even when additional training is considered, models such as Qwen-Image-Edit already achieve strong content preservation while supporting arbitrary image inputs. The proposed approach seems closer to modifying the conditioning halfway through generation. If the goal is simply to generate an image under the modified condition, why not generate directly using the target prompt? If preserving the original content is essential, what advantages does this approach offer over existing image editing methods?

How does the underlying idea fundamentally differ from existing prompt travel or timestep-dependent prompt switching techniques?

Reviewer_zSnJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces "Dancing Diffusion," a technique that modifies the conditioning input to a diffusion model during denoising. The authors demonstrate that his approach works for image editing and allows for controllable modification of where objects are being changed.

Critical strengths

The overall method is well-defined and successfully conveys the use of modifying the conditioning into a diffusion model during generation.

Critical weaknesses

I have not run this paper through an AI generation detection tool. The vast majority of this paper reads as entirely AI-generated. Regardless, the technical content is extremely difficult to interpret beyond the main method.

The goals for evaluations are extremely difficult to understand. The key results in this work first demonstrate controllability of generation location on toy images. The goal of the results is difficult to interpret, e.g. "Exact-mask subject and background MSE on 1,024 validation images, normalized pointwise by the single-expert MSE, so that one denotes parity and lower is better." What is 'one'?. Also figure captions are weird " Figure 4: Editing with visible scene retention. Genuine, uncropped outputs for the same source and request: final checkpoint, schedule."

The results in this paper are known to the community and noise levels being used to disentangle image editing are a classic technique.

The qualitative results are not obviously better. I see no perceptual gains, and if there are any they should be indicated to the reader.

Questions for authors

Ask only questions whose answers could materially affect your assessment. Distinguish clarification requests from suggestions for future work.

Why is introducing a trajectory that dances necessary?

Will the authors compare against image editing baselines?

Comments on the paper

1 comment

Hi authors,

If you are planning to respond to the reviews, I am happy to receive your responses to my concerns in stages. This would also allow us to engage in further discussion and clarification, given that only 1 day remain before the discussion phase ends.

Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.