acceptodds
Withdrawn from ICLR 2027

PIT: Preserving Identity through Trajectories for Layout-Aware Multi-Reference Image Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Layout-aware multi-reference image generation aims to place multiple reference subjects in prescribed regions while preserving their identities and adapting them to a coherent text-specified scene. Recent methods strengthen reference-to-region binding through layout-conditioned inputs or structured attention, typically relying on a single evolving latent trajectory for both identity preservation and scene adaptation. However, reference-specific evidence may become less recoverable from the evolving scene-adaptive state, even though reference conditioning remains available at every step. Once identity deviations arise, they may persist across subsequent steps, while no alternative identity history is retained for correction. To address this limitation, we propose PIT, a framework that maintains a persistent identity trajectory alongside a primary trajectory. Both trajectories share the same text and reference conditions but evolve under different guidance rules. Accordingly, the primary trajectory adapts the generation to the target scene, whereas the identity trajectory maintains an alternative state history with greater emphasis on reference guidance. To exploit this alternative history, PIT compares the two trajectory states at the same timestep and uses the difference between their clean predictions to correct the primary trajectory within the estimated subject regions. Extensive experiments show that PIT achieves a strong overall balance across identity preservation, spatial accuracy, and text consistency.

Primary area:
generative models
Keywords:
Multi-Reference Image Generation, Layout-Guided Image Generation, Subject Identity Preservation, Diffusion Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Yv1WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PIT for layout-aware multi-reference image generation, aiming to preserve subject identity while following spatial layouts and adapting subjects to text-specified scenes. After a shared warm-up, PIT maintains a primary trajectory and a reference-oriented identity trajectory. It corrects the primary trajectory within subject regions using the difference between their clean-latent predictions under the same identity predictor, while synchronizing the auxiliary trajectory’s surrounding context. The method is implemented with Qwen-Image-Edit-2511, layered reference inputs, and LoRA adaptation, and evaluated through benchmarks, ablations, multiple random seeds, and a user study.

Critical strengths
  1. The complete system is competitive on a practically relevant task. Multi-subject generation requires jointly satisfying identity, layout, and scene consistency. PIT achieves an overall AVG of 70.18 on LAMICBENCH++, compared with 65.27 for the strongest external baseline, and the user study also supports its overall effectiveness. These results demonstrate practical value beyond a conceptual modification to sampling.

  2. The paper provides meaningful internal controls supporting the effectiveness of the added mechanism. It includes a single-trajectory baseline sharing the adapted model, alongside ablations of warm-up, trajectory correction, and context tethering. PIT improves overall AVG across all three matched random seeds. This increases my confidence that the gains are reproducible, although their magnitude and computational efficiency require further justification.

Critical weaknesses
  1. The incremental contribution of the dual-trajectory mechanism is modest, and its benefit relative to the additional inference cost remains unclear. The single-trajectory baseline already achieves an overall AVG of 69.64, exceeding every external baseline. PIT increases this to 70.18, with gains of 1.90 in IDS and 0.25 in IPS. Meanwhile, generation latency increases from 41.61 to 65.68 seconds per image, approximately 58%. Thus, the external comparisons establish the competitiveness of the complete adapted system more clearly than the significance of the dual-trajectory contribution itself. Although the controlled baseline matches the model and sampling steps, it does not match the inference budget. The experiments therefore do not establish whether allocating additional computation to single-trajectory sampling or adjusting its guidance schedule could yield comparable improvements. This materially limits my assessment of the core contribution.

  2. The trade-off between identity preservation and scene adaptation is insufficiently evaluated. Some results in Figure 3 retain poses and appearance configurations closely resembling the references, giving a collage-like impression, the requested interaction is unclear. These examples do not establish that the dual-trajectory mechanism causes the issue, but they raise the possibility of reference over-preservation. The paper lacks paired single-trajectory/PIT comparisons for prompts explicitly requiring changes in action or pose, or interactions between subjects. Consequently, it remains unclear whether improved identity scores come at the expense of generative flexibility. Substantial overlap and occlusion are also insufficiently tested, while the union subject mask does not itself specify subject ownership in overlapping regions. These gaps weaken the central claim that identity preservation and scene adaptation are jointly maintained.

Questions for authors
  1. How does PIT compare with the single-trajectory baseline under the same runtime or model-evaluation budget? For example, if the additional budget is allocated to more single-trajectory sampling steps, does PIT still offer a better trade-off between identity preservation and text consistency?

  2. Could you provide paired single-trajectory/PIT results using the same model, references, layouts, and matched random seeds, focusing on prompts that explicitly require changes to the reference pose or action, as well as subject contact or partial occlusion? These comparisons would help determine whether the identity gains involve excessive preservation of reference appearance.

  3. Equation (17) aggregates individual subject supports into a union mask. When subject supports overlap, how does the method maintain correct reference-to-subject correspondence and handle occlusion? Are these capabilities primarily inherited from the backbone, or does the trajectory correction include an additional mechanism to prevent incorrect attribute transfer or preservation of subject parts that should be occluded?

Reviewer_21KTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies layout-aware multi-reference image generation. The authors argue that with a single denoising trajectory, subject identity can drift during sampling and is hard to recover afterwards. PIT runs an extra identity trajectory in parallel, which drops the text-guidance residual and keeps the reference guidance. At each step, the same identity predictor is applied to both states, and the difference of the two clean predictions is added to the primary prediction inside an attention-based subject mask. The method is built on Qwen-Image-Edit-2511 with a LoRA and evaluated on LAMICBench++, COCO-MIG and LayoutSAM-Eval.

Critical strengths
  1. The idea is simple: keep a separate identity trajectory and use it to correct the main one at inference time. PIT itself needs no extra training and can be plugged into a flow-matching sampler.

  2. The comparison with the single-trajectory counterpart is controlled (same LoRA, masks and seeds), and the improvement holds over three seeds. PIT also gets the best AVG on LAMICBench++ (70.18 vs. 65.27 for ContextGen).

Critical weaknesses
  1. Inference cost: latency increases from 41.6s to 65.7s per image (1.58x, Table 5). Could the identity trajectory be kept only for the early/middle steps to reduce this cost?

  2. Some results look unnatural. In Fig. 3 (second row), the hat/head layer in the layout is already too large for the dress, and PIT copies this proportion instead of adapting it, while e.g. MS-Diffusion gives a more natural body.

  3. Multi-person interaction is not discussed, and no such examples are shown. The correction mask M_t is a single union over all subjects rather than per-subject (A.2, Eq. 17), so it is unclear how PIT avoids identity mixing when people overlap.

  4. Some baselines are outdated (e.g., MS-Diffusion uses SDXL). In addition, the single-trajectory version already reaches 69.64 AVG, above all baselines, while PIT adds only ~0.5. So most of the gain seems to come from the backbone, LoRA and layered input rather than the dual-trajectory correction.

Questions for authors

Please see Weaknesses. Moreover, with many subjects the union mask covers much of the image. Does the correction then affect the background or text alignment? Separate results on the >5 subject cases would help.

Reviewer_Vf82OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PIT, a dual-trajectory inference framework for layout-aware multi-reference image generation. PIT maintains an auxiliary identity trajectory alongside the primary generation trajectory. The two trajectories use different guidance rules, and their matched clean-prediction difference is used to correct the primary trajectory within estimated subject regions. The method also introduces identity-aware warm-up and context tethering to balance identity preservation and scene adaptation.

Critical strengths
  1. The paper addresses a meaningful and practical problem. Preserving reference-specific identities while adapting multiple subjects to a coherent scene is challenging and relevant to personalized image generation. The observation that correct reference-to-region binding does not necessarily guarantee identity preservation is reasonable and motivates further investigation.
  2. The proposed framework is clearly structured and reasonably motivated. The separation between a scene-adaptive primary trajectory and an identity-oriented auxiliary trajectory provides an intuitive way to balance the two competing objectives.
  3. Experimental evaluations and ablations are comprehensive.
Critical weaknesses
  1. The main contribution of PIT is its dual-trajectory architecture. However, according to Tables 3 and 5, PIT improves the overall AVG by only 0.54 points over the single-trajectory baseline, while increasing inference time from 41.61 to 65.68 seconds. Could the single-trajectory baseline achieve a comparable improvement simply by increasing the number of inference steps to match PIT's computational cost?
  2. The paper claims that reference-specific information becomes less recoverable during single-trajectory generation. However, the single-trajectory similarity curve in Figure 1(d) does not clearly demonstrate progressive identity degradation. In my view, this appears more like insufficient identity fitting rather than a gradual loss of previously preserved identity information. I wonder whether gradually increasing the reference-image guidance weight over denoising timesteps could improve the single-trajectory similarity curve and achieve comparable identity preservation.
Questions for authors
  1. Table 4 shows that identity-aware warm-up has a substantial influence on performance. Can the authors provide a corresponding warm-up-length study for the single-trajectory baseline?
  2. Appendix A.2 states that the dynamic subject-support masks are constructed from attention maps collected during the evaluation of the identity trajectory. However, the controlled single-trajectory baseline does not maintain this auxiliary state. How does the single-trajectory baseline obtain its dynamic masks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.