acceptodds
Under review as a conference paper at ICLR 2027

CoPlace: Joint Subject Placement and Empty Scene Planning for Sequential Image Generation

Abstract

Multi-modal Procedure Planning (MPP) generates textual and visual steps towards accomplishing a specific goal. While recent developments in large language models have empowered effective textual reasoning and planning, generating coherent visual procedures remains challenging due to the requirements of subject consistency, state transition coherence, and subject-background harmony. Existing methods typically treat these problems separately, leading to inevitable trade-offs. To address these challenges, we propose Transformation, Plan then Place (TPP), a novel multi-stage framework for sequential image generation. TPP decomposes the generation process into three stages: subject state transformation, joint subject layout and empty-scene planning, and subject placement. We also introduce CoPlace, a joint generation model, where layout planning and scene generation iteratively constrain each other during the denoising process. Furthermore, we propose a space-gating mechanism that suppresses text leakage within subject placement regions, encouraging the generation of a semantically empty subject placeholder that is compatible with global scene context. Extensive experiments on HD-EPIC, LayoutSAM, and HiCo-7K demonstrate that our methods achieve superior performance in improving layout planning, image quality, and geometric compatibility between subjects and backgrounds. Our approach provides a flexible solution for generating coherent multi-step visual procedures while effectively balancing subject fidelity, text alignment and scene controllability.

Primary area:
generative models
Keywords:
Multimodal Procedural Learning, Sequential Image Generation, Customized Image Generation, Conditioned Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EfxPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Transformation, Plan then Place (TPP), a framework for sequential image generation that aims to preserve subject identity while allowing state changes and natural interactions with surrounding scenes. Its main contribution, CoPlace, jointly predicts subject placement and generates an empty background through iterative communication between a layout branch and an image generation branch. It also introduces a space-gating mechanism to prevent unwanted content in the reserved subject region. The framework uses Concept Sliders for subject transformation and AnyDoor for final placement. Experiments on three datasets report improvements in image quality, subject consistency, and subject-scene compatibility, alongside qualitative examples of multi-step procedural generation.

Critical strengths
  1. Coupling the layout branch (LB) with the image generator is an interesting idea. Where the subject should go plausibly depends on the scene, and the scene on where the subject goes, so predicting the box and rendering the background jointly seems better motivated than predicting layout first and rendering afterwards.

  2. Reciprocal Evolution (SE) is a good design choice. After the first 20% of the denoising steps, the layout branch and the empty-scene branch start exchanging information: the layout branch predicts the bbox (with a confidence score) from the current preview scene, the scene branch is conditioned on that bbox, and every 10% of the denoising process the bbox and the preview scene are updated and exchanged again. In this way the placement and the background progressively adapt to each other instead of one being fixed before the other. While iterative refinement between two components is not new, it is a sensible way to couple the two branches here.

Critical weaknesses
  1. The paper claims improvements in sequential visual generation (subject consistency, state transition coherence; contribution 4), but there is no evaluation of it. Tables 1 and 2 evaluate single images with a single object, and sequences appear only in two qualitative examples (Fig. 4, against TIP and MMP) with no metric.

  2. Important baselines are missing. I would prioritize two: (a) For final image generation: PlanGen, then inpainting to remove the object, then AnyDoor to add it back. The removal mask can be obtained automatically with SAM3, so the baseline needs no manual annotation. It differs from the included PlanGen+AnyDoor because it involves no PlanGen fine-tuning. The authors themselves build the empty-background ground truth by cropping subjects and inpainting with Big-LaMa (l.322-323), so generate-then-remove is essentially their own data pipeline. (b) A multi-reference image model such as FLUX.2 or Qwen-Image-Edit, given the reference subject (or its edited version) and the step description, and asked each time to generate the full image. Why can such a model not do the same editing, i.e., apply the state change to the subject and place it in a fitting background, without Concept Sliders and the bbox/empty-scene planning? In CoPlace the bbox comes from a VLM rather than the user, but my intuition is that these models already place the object sensibly while generating the background. If CoPlace's advantage appears mainly in long sequences, this should be shown with more visual examples. Optionally, commercial models (GPT Image, Nano Banana) given the reference or edited reference would be a useful reference point on the single-object experiments.

  3. The qualitative evidence is too weak. The number of qualitative examples is low, the image quality is noticeably below current models (likely in part due to the SD 1.5 backbone of the empty-scene branch, l.366), and Fig. 5a is very small. Some baseline inputs are unfair: the reference labelled "fishing boat" actually shows a person, so it is not a fair input to IP-Adapter. In Fig. 5b the reference labelled "older couple walking a dog through the woods" shows a single person. The SG and SE ablations are only quantitative (Table 2); there are no visual with/without comparisons (Fig. 5b only visualizes the SE evolution).

  4. Several claims are unexplained or not supported.

  • In CFG, the unconditional branch is not a way of removing objects: it still generates objects, just not ones guided by the text. So using it inside the box (Eq. 7) does not make the region "empty", and the paper needs either an intuition for why it should be empty or a visual ablation showing it (e.g., side-by-side examples of standard CFG vs. space-gated CFG).
  • Missing ablations: the spatially weighted loss (Eqs. 13-15) and the confidence head (Eq. 12), and how much each helps.
  • The user study has only 5 raters.
  • The training time of the Concept Sliders (per concept) is never discussed, although the state transformation depends on them.

The story would be easier to follow if SCRM were simply described as Concept Sliders for preserving identity while applying the edit, and SPM as AnyDoor for the final generation (as in the CoPlace+AnyDoor evaluation), which would also reduce the number of components a new reader has to track.

Questions for authors
  1. Why are the TPP and MMP step-1 images in the Fig. 4 pan-frying example identical?
  2. Why does the space-gated CFG still produce objects inside the reserved region in Fig. 5b, given that the SG is meant to remove objects?
  3. In Fig. 5b, the reference "older couple walking a dog through the woods" shows a single person. Is this intended?
  4. Can you provide an evaluation of the sequential setting (multi-step subject consistency and state-transition coherence), given that it is the main claim?
  5. How does CoPlace compare with (i) PlanGen followed by SAM3-based object removal and AnyDoor, and (ii) a multi-reference model such as FLUX.2 or Qwen-Image-Edit given the reference or edited reference? Commercial models (GPT Image, Nano Banana) would be a useful optional reference.
  6. What is the training time of the Concept Sliders per concept?
  7. What is the contribution of the confidence head (Eq. 12) and of the spatially weighted loss (Eqs. 13-15)? Please add ablations, and visual examples for the SG and SE ablations.

Overall, I lean towards rejection. I will read the other reviews and the rebuttal and take them into account, but I expect the main concerns would need more than a rebuttal to address, so a change in my assessment is unlikely.

1 comment

reviewer 3: Thank you for your constructive comments. We sincerely apologize for overstating Contribution 4 without sufficient experimental validation. We will revise the contribution statement accordingly and provide additional experimental results in the multimodal procedural learning scenario in our future work. We will also report a full combination of Spacial Gating Mechanism and Spiral Evolution ablation test and discutions in future submission. Reply to Q1: We apply TPP only when the step text is detected to mention the reference subject. Otherwise, we follow MMPlanner and generate the image using a prompt refined by an LLM through chain-of-thought reasoning. Reply to Q2: Our main concentration rest on placing the main subject complex background with various objects, which is a common scene in multimodal procedure setting, thus we aim to soly prevent the main subject appearance inside the reserved region. We will consider replace the term "empty" with other statement to prevent misleading. Reply to Q3: Due to automatic postpocessing, the dataset might be quite noisy and not strictly semantically aligned. Considering that it is labor intensive, we will try to carlibrate the dataset manually if possible in future work. Reply to Q6: Phragmatically, train one slider from 3-5 pairs of contrasting images cost approximately 3-5 minites. Reply to Q5 and Q7: We acknowledge these questions and will provide a more detailed technical discussion in our future work.

Reviewer_r9mrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets the visual steps in multi-modal procedure planning, where the same subject should stay consistent across steps, change state over time, and fit naturally into each scene. It proposes TPP, a three-stage pipeline. A state module based on Concept Sliders first changes the subject's state. CoPlace then generates an empty scene and a box for the subject at the same time. Its layout branch is built on PlanGen and its scene branch on Stable Diffusion 1.5, and the two exchange the current box and scene preview during denoising. A space gating rule removes text guidance inside the box so that the region stays free of objects. Finally, AnyDoor inserts the subject into the box. Training data is built from LayoutSAM, HiCo-7K and HD-EPIC by cropping subjects and inpainting the background. The experiments compare full-image and empty-scene generation against IP-Adapter, Lay-Your-Scene and PlanGen variants. They also include an ablation of the two mechanisms, a small human study, and two qualitative procedure examples.

Critical strengths
  1. Planning the placement together with the scene is a sensible idea. Where a subject should go depends on what is around it, yet most layout-first pipelines fix the box before the scene exists. Letting the layout branch and the scene branch update each other during denoising is a reasonable way to address this. Figure 5b gives an intuitive picture of how the box and the scene settle together.

  2. The decomposition handles the fidelity and text alignment trade-off in a practical way. The pipeline generates an empty scene with a reserved slot and leaves subject rendering to an insertion model. This keeps the subject close to the reference while the text controls the scene. The parts are modular and light enough to train on a single GPU, and the human study prefers CoPlace over the two pipeline baselines on all three criteria.

Critical weaknesses
  1. The sequential setting is not evaluated. The title, abstract and introduction frame the work as multi-modal procedure planning, where the subject should stay consistent and change state across steps. The only evidence for this setting is two qualitative examples in Figure 4. There is no multi-step test set, no cross-step consistency or state metric, and no comparison with consistent sequence or story generation methods. The state module gets one short paragraph without training details. Section 3.2 also says it is a fine-tuned PlanGen, while Section 3.3 says it uses Concept Sliders. As it stands, the experiments are about single-image subject placement, not procedure planning.

  2. Layout quality is never measured directly. The central claim is that joint planning gives better placement, yet no result evaluates the predicted box itself, such as IoU with the ground truth box or a human rating of placement plausibility. DINO-S and CLIP-S compare box features, but CoPlace+AnyDoor and PlanGen+AnyDoor both insert the same reference subject with AnyDoor, so these scores mostly reflect AnyDoor. LeakageScore is also unclear, since it is defined as a cosine similarity between two boxes.

  3. The main results do not clearly support the claims. In Table 1, CoPlace+AnyDoor is close to PlanGen+AnyDoor on most metrics and never leads on DINO-S. It has lower CLIP than PlanGen on HiCo-7K and much worse FID than PlanGen+AnyDoor on HD-EPIC. In Table 2, the tuned PlanGen has clearly better FID on all three datasets and better Overlay and Leakage on HD-EPIC. The abstract's claim of superior layout planning, image quality and geometric compatibility goes beyond what the tables show.

  4. The ablation is inconsistent and incomplete. The text says full CoPlace is obviously better on Overlay and Leakage. However, on HiCo-7K, removing both mechanisms gives a much lower Overlay, and on HD-EPIC the differences are very small. Reciprocal evolution is only removed together with space gating, so its own effect is not isolated. There is no ablation of the spatially weighted loss, the confidence head, or the exchange schedule. All results come from a single run without variance.

  5. The space gating design needs more justification. Inside the box, the model falls back to the unconditional prediction. Unconditional generation is not the same as an empty region, and it is not obvious why this gives a surface that suits the subject rather than arbitrary content. The training targets come from LaMa inpainting, which often leaves smooth or blurred patches, so the model may simply learn to produce such patches inside the box. An analysis of what the reserved regions look like, and of how AnyDoor behaves on them, would help.

  6. Baselines and the human study are limited. IP-Adapter and Lay-Your-Scene do not address subject placement in the same way. A natural baseline is missing: generate the scene from text, predict a placement with an object placement model such as TopNet or GracoNet, then insert the subject with AnyDoor. The related work also leaves out object placement and consistent sequence generation. The human study uses five undergraduate raters with a three-way forced choice, reports no agreement, and leaves out IP-Adapter, the strongest fidelity baseline.

  7. The writing needs substantial revision. Section 3.5 contains a broken symbol, the diffusion loss is cited as "Eq. equation 18", and template instructions are left in the AI use statement. The same mechanism is called Space-Gating, Spatial Text Gate and spatial gating. Reciprocal evolution is also called spiral evolution, and MMPlanner appears as MMP. Figures 3 and 5 are hard to read even when zoomed in.

  8. The scene branch is built on Stable Diffusion 1.5. This backbone is several generations behind current text-to-image models, and the authors themselves attribute the FID gap to PlanGen's larger capacity. It is therefore hard to tell whether the gains from joint planning hold on a modern backbone. I would like to see results with FLUX as the scene branch in the rebuttal, at least for the empty-scene comparison and the ablation in Table 2. Since FLUX.1 dev is guidance-distilled and has no separate unconditional pass, please also explain how space gating is implemented in that setting.

Questions for authors
  1. Can you add a quantitative evaluation on real multi-step procedures, with cross-step subject consistency and state correctness, and compare with consistent sequence generation methods? At the moment the procedure planning claims rest on two examples.
  2. How accurate is the predicted box itself? Please report IoU with ground truth boxes or a human rating of placement. Please also compare with a pipeline that generates the scene from text, predicts the placement with an object placement model, and then uses AnyDoor.
  3. How exactly are LeakageScore, DINO-S and CLIP-S computed? Since both CoPlace+AnyDoor and PlanGen+AnyDoor insert the subject with AnyDoor, which part of these scores reflects placement quality rather than insertion quality?
1 comment

Thank you for your constructive comments. We sincerely apologize for overstating Contribution 4 without sufficient experimental validation. Reply to Q1: We will revise the contribution statement accordingly and provide additional experimental results in the multimodal procedural learning scenario in our future work. We will also report a full combination of Spacial Gating Mechanism and Spiral Evolution ablation test and discutions in future submission. Reply to Q2: As the empty background generation is an open task, in our consideration, the positions of the subject placement area should not be constrained by the ground boxes. Instead of the accuracy of the predicted box, our concentration rest on the spacial-semantic overlap between the predicted subject placement region and objects detected in the generated empty background, which can be reflected by OverlayScore. We will consider your proposed baseline in furture work. Replay to Q3: We acknowledge the question and will provide a more detailed technical discussion in our future work.

Reviewer_YeWdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Transformation, Plan then Place (TPP), a modular pipeline for generating procedural image sequences while preserving a reference subject. It separates subject-state editing, joint layout and empty-scene generation, and subject insertion. Its main technical component, CoPlace, couples a PlanGen-based layout predictor with a Stable Diffusion 1.5 scene generator: intermediate scene previews inform bounding-box updates, which subsequently influence denoising. Spatially gated classifier-free guidance and a region-weighted diffusion loss encourage suitable insertion regions, and AnyDoor performs subject placement. Experiments on LayoutSAM, HiCo-7K, and HD-EPIC assess individual images, empty scenes, and human preferences. Two qualitative cooking examples illustrate the complete sequential pipeline. The results suggest a useful composition method, but the evidence for procedural coherence and the specific benefit of reciprocal refinement remains incomplete.

Critical strengths
  1. The joint placement and empty-scene formulation addresses a meaningful compositional problem. Reserving a region without considering its surroundings can produce inappropriate scale, support, or occlusion. CoPlace allows the evolving scene to inform placement and allows placement to influence subsequent scene generation. Separating this process from subject insertion is a sensible design that makes the contribution useful beyond the particular state-editing module chosen for TPP.
  2. The image-level results provide evidence of a useful fidelity/alignment trade-off. The human study also reports the highest selection rates for the proposed method across all three criteria and datasets.
Critical weaknesses
  1. The evidence does not establish the central sequential-generation claims. Tables 1, 2, and 4 evaluate individual images or empty scenes; Section 4.3 provides only two qualitative procedural examples. There is no quantitative evaluation of state-transition correctness, identity preservation across a complete sequence, or consistency of the surrounding scene across steps. Equations (2)-(4) condition each step on the reference and its current description without specifying an explicit dependency on previous generated steps. This does not preclude coherent sequences, but it makes direct sequence-level validation essential. The SCRM implementation discrepancy further obscures how the claimed transformations are obtained. Consequently, the results support a narrower subject-composition contribution more convincingly than the paper's claims about coherent visual procedures.
  2. The experiments do not adequately isolate the main technical contribution, and several conclusions overstate the reported evidence. Table 2 includes the full model, removal of space gating, and removal of both gating and reciprocal evolution. It lacks the complementary setting that retains gating while disabling reciprocal evolution, so the benefit of feedback in the complete system is not isolated. The main PlanGen comparison also changes the scene-generation backbone. A matched one-pass baseline, a full two-factor ablation, and evaluation after the same AnyDoor insertion stage are needed to attribute the gains to CoPlace's proposed interaction.
Questions for authors
  1. Which model actually implements SCRM? If it is Concept Sliders, how are the relevant sliders trained or selected, and how is the input subject preserved during editing? Can you provide sequence-level results for state correctness and identity consistency, with the compared methods receiving equivalent reference information?
  2. With the same training data, scene backbone, insertion module, and sampling settings, what happens in all four combinations of space gating on/off and reciprocal evolution on/off? Please report final-image and placement-quality results, alongside inference time. This would determine whether repeated feedback offers a benefit beyond gating alone.
  3. How are bounding-box coordinates and confidence scores injected into the diffusion model? What preview images does the layout branch receive during training and before the first feedback update?
  4. OverLayBench already defines OverLayScore as a sum over overlapping box pairs, weighted by CLIP-caption similarity. Which boxes and captions are used here, and how does this layout-complexity measure establish subject exclusion in an empty region?
1 comment

Thank you for your constructive comments. We sincerely apologize for overstating Contribution 4 without sufficient experimental validation. We will revise the contribution statement accordingly and provide additional experimental results in the multimodal procedural learning scenario in our future work. We will also report a full combination of Spacial Gating Mechanism and Spiral Evolution ablation test and discutions in future submission. Reply to Q1: In this paper, we implement SCRM using the classical Concept Sliders introduced in 2023, which can be trained from a few pairs of images or prompts representing contrasting semantics. Each semantic slider needs to be trained independently. More recent architectures, such as All-in-One Slider, can also be readily adapted to our MPP framework. Reply to Q2, Q3 and Q4: We acknowledge these questions and will provide a more detailed technical discussion in our future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.