acceptodds
Withdrawn from ICLR 2027

RefRoute: Decoupling Conditioning Cost from References via Compact Residual Conditioning and Spatial Routing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-reference image generation requires preserving the appearance of multiple subjects while composing them into a coherent scene. However, existing diffusion transformers commonly encode references as dense visual token grids and jointly process them with global attention, making conditioning increasingly expensive as the number and resolution of references grow. We present RefRoute, a framework that addresses both reference representation cost and attention overhead through two complementary mechanisms. Compact residual conditioning combines low-resolution latent tokens with lightweight residual features extracted from full-resolution pixels, reducing reference token counts while retaining fine-grained appearance cues. Condition routing and attention routing align reference tokens with their assigned target regions and restrict cross-reference interactions, while allowing selective reference access beyond region boundaries for scene integration. We further introduce RefRoute-Data for training many-reference generation models and ManyRef100, a benchmark spanning human, object, and mixed compositions with 10–17 references. After many-reference fine-tuning, RefRoute achieves an overall Weighted-Ref-VIEScore of 36.06 on ManyRef100, compared with 8.88 for FLUX.2-Klein-9B. Separate inference-cost evaluations show substantially slower latency growth as the reference count increases: at 16 references, our 50-step and 4-step configurations achieve 18.3× and 14.2× speedups over their corresponding FLUX baselines, respectively. These results establish compact reference representations and spatially routed attention as an effective approach to scalable many-reference image generation.

Primary area:
generative models
Keywords:
multi-reference image generation, controllable generation, diffusion transformers, visual conditioning, attention routing, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RrAxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors proposed RefRoute, an efficient framework for multi-reference image generation that reduces inference latency and peak memory usage as the number of visual references increases, while achieving competitive Weighted-Ref-VIEScore results. Specifically, the RefRoute is composed of the compact residual conditioning, condition routing and attention routing. Compact residual conditioning helps reduce the visual tokens input to DiT while maintaining high-frequency details of references. Condition routing uses separate reference and spatial LoRA adapters for condition-specific adaptation while attention routing helps explicit reference-region correspondence. Moreover, the authors devised the RefRoute-Data and ManyRef100 for training and evaluation for the multi-reference image generation task.

Critical strengths
  1. The innovation of the paper are concise and straightforward, and the experimental results claimed in the paper are promising and seem to be sound. Extracting tokens from downsampled reference images and compensating for the lost high-frequency details with a pre-trained residual prediction module provides a compression strategy that explicitly considers reconstruction quality. Meanwhile, the attention routing reduces unnecessary computation costs.
  2. The figures in the paper are easy to understand and clear. Figure 5 convictively presents the effectiveness of dynamic top-1 routing compared to strict spatial masking from qualitative comparison perspective.
Critical weaknesses
  1. The writing of experiments is kind of disordered. Line 369 and 370 claim that the authors examined both inference cost and generation quality as the number of references increases. However, in the main paper, I can not find the examination of generation quality as the number of references increases. If I misunderstand, I hope the author will correct me during the rebuttal.
  2. The experiments in the main paper are not that sufficient. The authors only compared the proposed method with general multimodal generation models that not specifically devised for multi-reference generation in Table 1 and 2 without comparing task-specific methods like StructGen or MultiRef mentioned in RelatedWorks. In the ablation study, Table 4 only confirms the effectiveness of stage 1 pretraining. However, I would like to see the impact of attention routing trained in stage 2 as well, such as its impact on the evaluation metrics, computation cost or inference latency.
Questions for authors
  1. Which part of the main paper is the mentioned examination of generation quality as the number of references increases?
  2. The reason of not comparing with the methods like StructGen or MultiRef mentioned in RelatedWorks.
  3. How to choose the top-k ranked reference. What is the choosing principle.
  4. Is corresponding target-region of reference not accessible during inference?
  5. Will the access of the top-k reference for the outside mask region increases the probability of generating the unnatural associations of references with the background, such as the man standing on the blue box with one foot in Figure 5?
Reviewer_6TboOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RefRoute, a framework designed to improve the computational efficiency of multi-reference image generation. The core of RefRoute consists of two complementary mechanisms: Compact residual conditioning and Condition and attention routing. The authors also construct RefRoute-Data, a training dataset with instance-level reference-to-spatial annotations, and ManyRef100, a 100-case evaluation benchmark with 10–17 references per sample spanning human, object, and mixed compositions. Built on FLUX.2-Klein-9B, RefRoute achieves large latency speedups and substantially outperforms baseline FLUX and closed-source commercial systems on the many-reference ManyRef100 benchmark.

Critical strengths
  1. The paper correctly identifies two distinct sources of inefficiency in multi-reference generation and proposes paired solutions that address both. The combination of fixed-size compact latent tokens with a pixel residual pathway is a sensible way to retain fine appearance information without inflating sequence length, and spatially routed attention explicitly enforces reference-region correspondence, which should mitigate identity leakage and attribute misattribution errors common in unstructured concatenation approaches.

  2. The inference cost results are compelling. At 16 references, RefRoute delivers an 18.3× speedup at 50 sampling steps and nearly flat peak memory growth relative to the dense FLUX baseline.

  3. ManyRef100 fills a gap in existing evaluation suites by focusing specifically on compositions with 10+ references paired with instance-level spatial assignments. Most existing benchmarks top out at 3–5 references, so this dataset enables more systematic evaluation of models in the many-reference setting.

Critical weaknesses
  1. The paper only compares against vanilla FLUX.2-Klein-9B, an OminiControl2 reference-branch implementation with an untrained adapter, and closed-source commercial systems (GPT-Image, Gemini). It entirely omits comparisons with numerous recently proposed, directly comparable open-source multi-reference generation methods, including Kaleido, RefAlign, StructGen, UniCustom, MOSAIC, and XVerse, among others. These methods all target the same multi-subject / multi-reference composition problem, and several already employ compact representations or spatial attention mechanisms conceptually similar to RefRoute.

Without comparing against these competitive, purpose-built open-source methods, the paper cannot demonstrate that RefRoute offers any meaningful advantage over current state-of-the-art approaches. Performance gains relative to a naive concatenation baseline are expected and unsurprising; they do not establish the novelty or effectiveness of the proposed method relative to existing specialized solutions. This severely undermines the significance of the results.

  1. RefRoute underperforms the native FLUX.2 baseline on MICo-Bench. This indicates that the proposed efficiency gains come at a measurable cost to generation quality in the far more common few-reference setting. The paper frames this as an expected trade-off, but it significantly limits the practical utility of the method. Since most real-world use cases involve a small number of references, a method that only excels at very high reference counts is of narrow applicability. The paper also does not explore whether the quality gap can be closed with further training or architectural adjustments.

  2. The two core mechanisms are largely incremental combinations of existing ideas. Compact reference encoding via downsampled latents has been explored in multiple prior works on efficient conditioning, and pixel-level residual feature injection has close parallels in high-resolution image generation literature. Similarly, spatial attention routing, reference-region masking, and restricted cross-attention are all well-established techniques in multi-subject and layout-guided generation. The paper does not clearly articulate how RefRoute differs from or advances upon this large body of prior work, beyond combining them in the many-reference setting.

  3. Evaluation relies almost entirely on Weighted-Ref-VIEScore, a VLM-based metric. The paper does not report standard objective image quality metrics such as FID, LPIPS, or CLIP similarity, nor does it include human evaluation of generation quality or reference fidelity. VLM-based scores are known to correlate imperfectly with human perception and can be biased toward textual alignment over fine-grained visual quality. Additionally, ManyRef100 contains only 100 cases, many of which are synthetically generated rather than using real photographic references, which limits the generalizability of the benchmark results.

Questions for authors

Refer to weakness part.

Reviewer_WQ6ROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper combines compact residual reference representations with spatially structured attention routing to reduce computational overhead and improve reference-region association. The authors additionally introduce RefRoute-Data and ManyRef100 to support training and evaluation. However, the experimental evidence does not adequately isolate the effects of the proposed architectural components from additional training and data distribution differences. Moreover, the main quality improvements are demonstrated primarily on a small, synthetically constructed benchmark, while performance decreases on the existing MICo-Bench.

Critical strengths
  1. The existing DiT performs Token Concatenation on multiple Reference images, which will cause the Reference Sequence to increase with the number of inputs. The author clearly distinguished between two issues, this kind of problem classification is reasonable.
  2. Computational efficiency gains and Multi-reference graph data and benchmarks may have community value if they open-sourced.
Critical weaknesses
  1. The method lacks essential innovation. Its main technical ideas, like low-resolution conditional encoding, position alignment, conditional branch isolation, and efficient Attention, all have clear precedents in existing work (OmniControl2/EasyControl). I think the innovation that RefRoute truly needs to prove is how much independent value these new designs provide within the existing efficient conditional framework.
  2. Some experiments need to be improved. (1) The improvements on ManyRef100 are difficult to attribute to the proposed architecture because the model undergoes additional many-reference fine-tuning, whereas the reported baselines do not appear to receive matched training. I think a comparison using identical training data and optimization budgets is essential. (2) Attention routing's independent effect on reference binding and inference efficiency is unclear. The authors should provide controlled ablations separating routing from compression.
Questions for authors

According to Appendix A.6 OminiControl2 mechanism comparison mentioned, the OminiControl2 baseline uses native-resolution reference tokens (4096 tokens per reference), with compact encoding and cross-step feature reuse disabled. Since these are central efficiency mechanisms of OminiControl2, how can the reported 4.5×–6.5× speedups establish an advantage over existing efficient conditioning methods? Could the authors provide a comparison against a fully optimized OminiControl2 implementation under equivalent settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.