acceptodds
Under review as a conference paper at ICLR 2027

Resolving Spatial and Semantic Conflicts in Multi-LoRA Composition via Softmax Routing and Cross-Adapter Attention Conditioning

Abstract

Multi-LoRA composition activates several Low-Rank Adaptation (LoRA) modules simultaneously so that a single diffusion forward pass generates an image that embodies multiple personalized concepts at once. Existing methods either merge adapter weight deltas uniformly corrupting concepts that share spatial territory—or route noise predictions via hard per-pixel argmax, which discards non-dominant LoRA contributions and collapses identity consistency on overlapping pairs. We propose two complementary, training-free mechanisms. SR-LoRA (Spatially-Routed LoRA) replaces hard argmax routing with temperature-controlled softmax masks over per-LoRA cross-attention maps, smoothly blending noise predictions at each spatial position. CLAC (Cross-LoRA Attention Conditioning) runs a two-pass UNet forward where the anchor LoRA's cross-attention keys and values are blended into the secondary LoRA's attention, suppressing appearance leakage; an IoU-based overlap detector activates this conditioning adaptively across all pair types. On the ComposLoRA benchmark (240 compositions per domain, 10 seeds), SR-LoRA and FreeFuse reach equivalent CLIP-T (+4.3% over LoRA-C), while CLAC improves CLIP-I by +7.6pp over FreeFuse on reality (0.915 vs. 0.839) and +0.9pp on anime. The combination (Adaptive Fusion) achieves the highest CLIP-I on anime (0.875), validated with per-pair-type breakdowns, ablations, and N=3 triplet experiments.

Primary area:
generative models
Keywords:
LoRA, multi-concept generation, text-to-image diffusion, attention routing, personalization, softmax routing, cross-attention conditioning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fcGpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies training-free composition of multiple LoRAs for image generation. It proposes two mechanisms: SR-LoRA uses cross-attention maps associated with each LoRA’s trigger words to spatially weight their noise predictions, while CLAC uses the overlap between attention maps to decide whether to blend the anchor LoRA’s attention keys and values into another LoRA’s generation pass. The paper evaluates these mechanisms on Stable Diffusion v1.5 across several concept combinations and reports ablations, hyperparameter sensitivity, and inference cost.

Critical strengths
  1. The paper evaluates spatial routing and cross-adapter attention conditioning separately, which helps clarify the role of each component.
  2. The experiments cover reality and anime subsets and report results across several concept pair types, hyperparameters, and inference costs.
Critical weaknesses
  1. The evaluation lacks character+character compositions. The two-concept experiments mainly cover character+clothing, character+object, character+style, and character+background. When two characters appear together, identity mixing, missing subjects, and spatial conflicts may be more pronounced. The current experiments therefore do not sufficiently test the method in this important multi-subject setting. Related work such as OMG (ECCV 2024) evaluates man+woman, man+man, and woman+woman compositions, while LoRAShop also demonstrates compositions of multiple character LoRAs. I cite these works to motivate evaluating this setting, rather than to request additional numerical baselines.

  2. The citation and description of the main FreeFuse baseline are inconsistent with the cited work. The bibliography attributes FreeFuse to “Liang Jian et al., 2025,” whereas the linked paper, FreeFuse: Multi-Subject LoRA Fusion via Adaptive Token-Level Routing at Test Time, is by Yaoli Liu, Yao-Xiang Ding, and Kun Zhou. Moreover, this submission describes FreeFuse as hard per-pixel argmax routing of multiple UNet noise predictions and states that SR-LoRA recovers FreeFuse as the temperature approaches zero. This differs from the cited FreeFuse paper’s routing of LoRA residuals at the token level. Because FreeFuse is a main baseline, this discrepancy affects the interpretation of both the comparisons and the claimed methodological relationship.

Questions for authors
  1. The choice of attention layers lacks an ablation. The method aggregates subject attention maps from the UNet mid block and up_blocks.1, but does not compare other layers or layer combinations, or directly assess the localization accuracy of the resulting maps. Both SR-LoRA’s spatial routing and CLAC’s overlap decision depend on these maps, so the effect of this design choice warrants further analysis.
  2. Could the authors clarify exactly how the “FreeFuse” baseline was implemented? Does it reproduce the cited paper? If the experiment instead uses a hard-argmax noise-routing baseline constructed by the authors, please distinguish it from FreeFuse and revise the citation and the statement that SR-LoRA recovers FreeFuse as the temperature approaches zero. Please also check the accuracy of the other references.
  3. Could the authors provide quantitative results and visual examples for character+character compositions? In particular, it would be useful to assess whether both characters appear, whether each retains its identity, and whether their identities become mixed.
Reviewer_FUUAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a dynamic weighting approach for Multi-LoRA composition in image generation. Previous baselines naively merge weights or predicted noise, which can result in concept confusion or artefacts when multiple LoRA adapters are combined. This work proposes extending FreeFuse so that the hard argmax of LoRAs over latent space is replaced with a soft routing map.

Critical strengths

The idea of localisation is intuitive, and constructing a map using the text activations is a very good idea. The mask mechanism and its subsequent use in composition are well explained.

Critical weaknesses

The biggest issue is the selective inclusion/exclusion of baselines in this work. Key works on the same task and benchmark, such as CMLoRA, LoRAtorio, MultiLFG, are missing from the comparisons. This is a fundamental flaw of the paper, especially as the core spatial mechanism is extremely similar to LoRAtorio.

In addition, the paper limits the comparison to only two LoRAs, while the benchmark includes up to five. This is very problematic, as we do not see the robustness of the method to NN. Furthermore, no LLM-as-a-judge or human evaluation against key baselines is done.

One smaller issue, in practice, three small methods are proposed; it is unclear which one is the main method and what the ablation.

There is a factually incorrect statement in the intro. Line 39: LoRA-Composite does NOT average out the deltas; it averages the predicted noise. The average of deltas would be merge or soup.

Questions for authors
  • What is the main methodological novelty against LoRAtorio?
  • How does the method perform when using an LLM as a judge and human evaluations in a qualitative comparison?
  • How does the method perform for N={4,5}
  • SD1.5 is outdated; can you show the method works on more recent architectures? Maybe DiT and flow-matching?
Reviewer_CoDDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes two training-free mechanisms for multi-LoRA composition in text-to-image diffusion models. SR-LoRA uses cross-attention-derived spatial softmax weights to combine per-adapter noise predictions. CLAC blends an anchor adapter’s cross-attention keys and values into a secondary adapter, gated by attention-map overlap. Experiments on ComposLoRA report text alignment comparable to the FreeFuse baseline and increased similarity to LoRA-C reference outputs, alongside ablations and three-adapter experiments.

Critical strengths
  1. Training-free composition. The method requires no additional training or user-provided spatial layouts.

  2. Evaluation of component tradeoffs. Per-pair-type results and ablations expose useful tradeoffs, including cases where combining the modules reduces performance.

Critical weaknesses
  1. Unclear baseline definitions. The paper describes LoRA-C as single-pass weight merging, whereas the cited method aggregates separate per-LoRA predictions. The FreeFuse attribution and description also require clarification, making the novelty and fairness of the comparisons difficult to assess.

  2. Limited evidence for the claimed benefits. CLIP-I measures similarity to LoRA-C outputs rather than independent identity references, and the default CLAC setting underperforms disabled conditioning in its ablation. Given the incremental design, the experiments do not establish sufficiently strong benefits or explanatory insight to support acceptance.

Questions for authors
  1. Which implementations of LoRA-C and FreeFuse were used, and how do they correspond to the cited methods?

  2. Do the identity improvements hold when evaluated against independent concept references rather than LoRA-C outputs?

  3. Why does the default CLAC setting (α=0.5) underperform disabled conditioning in Table 6?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.