acceptodds
Withdrawn from ICLR 2027

When Unified Models Meet Visual Experts: Lightweight Collaboration via Generative Handoff

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Unified multimodal models (UMMs) offer a single, flexible interface for visual understanding, generation, and editing, but specialized image generators and editors often achieve better visual quality on the tasks they are built for. Getting the two to work together, however, typically requires additional alignment training, which is costly and can compromise the UMM's broader capabilities. We find that, for several UMM–expert pairs, their latent spaces and generative trajectories are already compatible enough for one model to continue sampling from a state produced by the other. Therefore, we introduce Handoff, a training-free collaboration strategy, where the UMM performs the early sampling steps, then passes its intermediate latent state directly to a visual expert, which completes generation from a selected timestep. Handoff retains the UMM's instruction-following and semantic capabilities while improving perceptual quality, without learned connectors or cross-model alignment. We further introduce Partial On-Policy Distillation (P-OPD) for settings where the expert is unavailable at inference time. P-OPD distills the expert's behavior only over the later sampling steps, using a timestep-gated LoRA, so that the resulting UMM can reproduce the benefit without calling the expert. Across image generation and editing tasks and multiple UMM–expert pairs, both methods consistently improve the underlying UMM. Handoff can even outperform both standalone models, including on the challenging LongText-Bench and CVTG-2K benchmarks. Together, these results show that UMMs and specialized visual experts can collaborate through their existing generative representations, with little or no additional alignment.

Primary area:
generative models
Keywords:
unified model, generative model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SyZ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Handoff: a unified multimodal model starts generating an image, and an image expert takes over to finish it. This requires compatible models that share the same VAE, but no additional training. It also proposes P-OPD, which teaches the unified model to imitate the expert during the later steps, so the expert is no longer needed at inference. Experiments cover image generation and editing across several model pairs. Handoff shows strong gains in text rendering and sometimes beats both models used alone. P-OPD improves BAGEL on all six benchmarks.

Critical strengths
  1. A simple method delivers clear improvements. Handoff needs no additional training and works across several model pairs. It can outperform both models used alone. For example, BAGEL→FLUX scores 0.655 on LongText-Bench, compared with 0.390 for BAGEL and 0.612 for FLUX. This is convincing evidence that the two models can help each other rather than merely relying on the stronger one.
  2. The distillation method makes the idea more practical. P-OPD improves BAGEL without requiring the expert at inference. The ablations support its two main choices: train only the later steps, and train on states produced by the current student. Together, these choices give more consistent gains than the tested alternatives.
Critical weaknesses
  1. The handoff timestep needs clearer justification and broader testing. The paper uses fixed handoff steps, but only studies timestep sensitivity for BAGEL→FLUX on LongText-Bench. It is unclear how the other handoff steps were selected and whether a separate validation set was used. This makes it difficult to judge whether the method works reliably across different prompts, resolutions, and sampling budgets without further tuning. Please explain the selection procedure and provide representative sensitivity results for editing and another model pair, including how the handoff step should change when the number of sampling steps changes.
  2. P-OPD is only validated on one unified model. Handoff is tested on three unified models, but P-OPD is tested only on BAGEL, with separate adapters for generation and editing. The broader Handoff results therefore do not establish that P-OPD also works across different unified models. Please evaluate P-OPD on another UMM, or clearly limit the conclusions to BAGEL and discuss what may affect its use with other models.
  3. The effects of partial training and late-only inference are not separated. Table 3 already compares full-trajectory OPD with P-OPD. However, this comparison changes both which steps are trained and when LoRA is enabled, so it does not show which change drives the improvement. Please take the same LoRA checkpoint trained with full-trajectory OPD and compare enabling it throughout sampling versus only after the handoff step, then compare both with P-OPD. This would help show whether keeping the original UMM in the early steps is sufficient, or whether training only the later steps provides an additional benefit.
Questions for authors

See weakness.

Reviewer_RvBaOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Unified multimodal models follow instructions well but render images worse than specialized experts. Observing that when the two share a VAE and a compatible flow-matching schedule, an expert can continue sampling from the unified model's intermediate latent, the authors propose Handoff: the unified model runs the early steps and the expert finishes the rest, without training. They also propose P-OPD, a LoRA active only in the later steps and trained on the unified model's own rollouts to match the expert, so the expert is not needed at inference.

Experiments pair BAGEL and OmniGen2 with FLUX.1-dev / Kontext and InternVL-U with Qwen-Image / Edit on four generation and two editing benchmarks.

Critical strengths
  1. The method is simple and training-free, and the finding that a text-to-image expert can complete an editing trajectory without access to the source image (Fig. 7) is interesting.
  2. Gains on text rendering are large, e.g., BAGEL→FLUX raises CVTG-2K from 0.356 to 0.596, above FLUX alone (0.497).
  3. The control experiments are informative, including the transferred-state variants (Table 5) and the 2×2 ablation of on-policy and partial training in P-OPD (Table 3).
Critical weaknesses
  1. Handoff is the training-free denoiser switching of T-Stitch applied to a UMM–expert pair. Since the method requires both models to share the same VAE and noise/time parameterization, their compatibility is expected, not a new finding.
  2. P-OPD is trained and evaluated only on BAGEL. Does it also work for OmniGen2 and InternVL-U?
  3. What are the inference latency and memory cost of Handoff compared with each standalone model?
  4. Sec. 4.1 specifies h = 15 for InternVL-U handoff to the text-to-image expert, but Table 2 has no InternVL-U Handoff† row and App. Table 4 / A.1 list only h = 4 and 9.
  5. Minor: duplicate references (OneIG-Bench 2025a/b, Step1X-Edit 2025a/b), and "h = 10 for training" in App. A.2 should read "for generation".
Questions for authors

see weakness

Reviewer_yAeJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Handoff, a training-free collaboration method between unified multimodal models (UMMs) and specialized visual models. The UMM performs the first part of the generation trajectory to establish semantics, layout, and editing intent, and then passes its intermediate latent state to an image-generation or image-editing expert for the remaining sampling steps. The method requires compatible VAE latents, noise parameterizations, and timestep schedules. The authors further introduce P-OPD, which uses timestep-gated LoRA adapters to distill the expert's velocity predictions only over the later part of the trajectory, allowing the UMM to operate without the expert at inference time. Experiments cover three UMMs, four experts, two VAE families, and both image generation and editing. The results show clear gains in text rendering and some editing metrics, but the improvements are not consistent across all model pairs, tasks, or metrics.

Critical strengths
  1. The motivation is relevant and practically meaningful. UMMs and specialized visual models have complementary capabilities: UMMs are generally better at complex instruction understanding, layout, and multimodal context, while specialized models often provide better visual details, text rendering, or editing quality. Having the former determine what to generate and where to place it, and the latter determine how to render it, is intuitive.

  2. The method is simple and easy to understand. Handoff does not require a cross-model connector or additional alignment training; it only switches the denoiser at an intermediate sampling step. P-OPD also trains only LoRA adapters while keeping the UMM backbone frozen.

  3. The evaluation covers several models and tasks, with notable gains in text generation. The paper evaluates three UMMs, four experts, and two VAE families. In particular, BAGEL→FLUX.1-dev improves LongText-Bench from 0.390 to 0.655 and CVTG-2K from 0.356 to 0.596, suggesting that the layout/semantic capabilities of the UMM and the rendering capabilities of FLUX may indeed be complementary.

Critical weaknesses
  1. The shared-VAE assumption substantially limits applicability. The method relies on compatible VAE latents, noise parameterizations, and timestep schedules. Therefore, the claim that independently trained models can directly collaborate should be stated more carefully. In practice, users would need to find a model pair with compatible VAE and sampling conventions; training new models from scratch would also require designing both models around the same VAE, which considerably restricts model choice and practical applicability.

  2. The complementary gain is not universal. BAGEL→FLUX.1-dev produces strong improvements on LongText-Bench and CVTG-2K. However, for the Qwen-Image-VAE family, InternVL-U→Qwen-Image Handoff only slightly exceeds Qwen-Image on OneIG-Bench Overall, while remaining substantially worse on LongText-Bench and CVTG-2K. The editing results also show trade-offs: handing off to a generation expert can improve perceptual quality while reducing semantic consistency. The paper should therefore distinguish more clearly between outperforming both standalone models on selected metrics and consistently combining the strengths of both models.

  3. The “lightweight” claim mainly concerns training and parameters, not inference cost. Handoff avoids additional alignment training, but inference still requires loading and running two large models. The paper does not report latency, memory usage and FLOPs, so it is unclear whether the method provides a practical deployment advantage.

  4. The comparison experiments are incomplete. The current experiments mainly compare UMM-only, expert-only, and Handoff. Important missing baselines include:

    • reverse switching, i.e., expert prefix → UMM suffix;
    • velocity interpolation;
    • a simple pipeline that sends the UMM-generated image to an expert for refinement;
    • explicit endpoint results for (h=0) and (h=K), together with complete handoff curves;

    These comparisons are needed to determine whether the gains come from the specific Handoff design or simply from the general effect of switching models along a trajectory. For the experiments mentioned above, it is best to include results from different model paris across multiple evaluation sets.

  5. The validation of P-OPD is narrow. P-OPD is mainly evaluated in the BAGEL→FLUX setting and is not demonstrated on OmniGen2 or InternVL-U. The generalization capability of this approach is unknown. Its gains may depend on a particular UMM–expert pair and training configuration.

Questions for authors
  1. If the two models use different VAEs, could the UMM intermediate state first be decoded into an image, re-encoded using the expert's VAE, re-noised to the corresponding timestep, and then passed to the expert for continued denoising? For example, would BAGEL→Qwen-Image or FLUX→InternVL-U be feasible in this way? How much reconstruction error and performance degradation would the decode–re-encode operation introduce?

  2. Beyond decode–re-encode, could the authors learn a lightweight mapping from the UMM latent space to the expert latent space? More generally, how robust is Handoff to differences in VAE normalization, timestep schedules, and noise parameterizations?

  3. The editing evaluation does not fully isolate source preservation and edit quality. The reported editing metrics are largely semantic or overall scores. In addition, FLUX.1-Kontext receives the source image directly, whereas FLUX.1-dev receives only the instruction and the transferred state. This makes comparisons between editing and generation experts asymmetric. The paper should separately evaluate source identity preservation, unchanged-region fidelity, edit localization, and changes outside the edited region.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.