acceptodds
Under review as a conference paper at ICLR 2027

Granularity-Aligned Selective Conditioning for Image Style Transfer

Abstract

Despite recent advances in reference-based image style transfer, existing methods still struggle to transfer global appearance and local stylistic details while preserving content structure. We introduce Granularity-Aligned Selective Conditioning (GASC), a framework that aligns style granularity with the distinct conditioning interfaces in a pretrained Diffusion–Mamba denoiser. GASC assigns shared image-level style conditioning to state-transition dynamics and token-varying style carriers to input writes. To construct these conditions, per-head orthogonal residualization reduces content interference in reference features, while content-conditioned carrier retrieval captures local style information. A separate content stream anchors spatial structure throughout denoising. Together, these designs coordinate global and local style conditioning with content preservation. Extensive experiments demonstrate that GASC outperforms existing methods in overall stylization quality while maintaining competitive content preservation.

Primary area:
generative models
Keywords:
Style Transfer, Diffusion Models, State Space Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PGvUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GASC proposes a style transfer framework based on the Diffusion-Mamba model. The motivation is that the global and local styles from a reference image should be injected into the model through different conditioning interfaces. Accordingly, the authors designed several modules to inject the global and local styles into the Δ\Delta and B parameters of the Mamba architecture, respectively, while preserving the image structure.

Critical strengths

GASC makes reasonable assumptions based on the various functions of the SSM interface and conducts a 2×2 granularity allocation study along with a single-route ablation. This approach achieves the best performance among the methods compared across the metrics, thereby demonstrating its effectiveness.

Critical weaknesses
  1. Although GASC proposes a sound and effective approach, its core technical concepts overlap with existing work. Its innovation lies primarily in the integration and refinement of prior methods.
  2. The authors have thoroughly validated the effectiveness of their proposed design through 2×2 granularity allocation, single-route ablation, and visual comparisons. However, these experiments primarily focus on the final generation quality and do not explicitly analyze how Δ\Delta and B individually influence global style versus local texture.
Questions for authors
  1. It would further strengthen the persuasiveness of the proposed granularity interface alignment hypothesis if the authors could provide more analysis of how conditioning Δ\Delta and B affects global stylistic details respectively.
  2. It would be interesting to compare the proposed group-level carrier retrieval with standard token-level retrieval to better understand the benefits of grouping.
  3. In the implementation details, the group queries are computed by sum pooling rather than mean pooling. So the magnitude of the queries will be changed accordingly. It would be helpful to clarify the motivation behind this design choice.
  4. Although C2RC^2R aims to avoid relying on predefined spatial correspondence, the final carrier fusion incorporates spatially aligned raw reference features. It would be helpful to clarify the motivation behind this design and how it affects style transfer when the content and reference images have different spatial layouts.
1 comment

I thank the authors for their detailed rebuttal and additional experiments.

Regarding W2, Q1, Q2, and Q3, the authors have addressed my concerns through supplementary experiments and explanations.

My remaining concerns are primarily related to novelty and spatial correspondence. Regarding novelty, although the authors have further clarified the distinctions between GASC and existing methods, I still believe that the contribution primarily lies in the effective integration and refinement of existing techniques. Regarding spatial correspondence, while the authors provide a reasonable justification for incorporating raw reference features and a dynamic gate, the potential impact of spatially aligned raw reference features when content and reference images have different spatial layouts has not been directly evaluated.

Overall, the rebuttal has addressed most of my concerns. I have decided to maintain my original rating of Weak Acceptance. I also encourage the author to upload a revised manuscript before the end of the rebuttal period.

Reviewer_bEYiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies arbitrary reference-based image style transfer and proposes Granularity-Aligned Selective Conditioning (GASC), built on a pretrained Diffusion–Mamba denoiser. The method injects image-level style conditions into the Δ\Delta pathway and token-varying style conditions into the BB pathway. It further incorporates per-head orthogonal residualization (HOD), content-conditioned carrier retrieval (C²R), and a separate content stream. Experiments report improved FID, ArtFID, and CFSD compared with existing methods, alongside a trade-off between stylization and content similarity.

Critical strengths
  • Coherent architecture. The framework explicitly separates image-level style, spatially varying style, and content structure, and specifies their respective conditioning pathways. This provides a meaningful approach to coordinating appearance transfer and content preservation.

  • Competitive empirical performance. GASC achieves the lowest FID, ArtFID, and CFSD among the evaluated methods in Table 1. The qualitative examples also demonstrate successful palette transfer and recognizable content structure across several styles, supporting the practical value of the complete framework.

Critical weaknesses
  1. Insufficient justification of the Diffusion–Mamba formulation and its incremental contribution. GASC inherits the pretrained DiM backbone and uses a standard diffusion training objective. Meanwhile, Mamba-ST already derives BB and Δ\Delta from style features and CC from content features. Although SaMam is included in Table 1, the closely related Mamba-ST and the cited StyMam are not evaluated. More importantly, there is no comparison with simpler or existing conditioning designs implemented within the same DiM backbone and training budget. Consequently, the experiments do not adequately separate the contribution of the proposed conditioning from the pretrained generative backbone and paired supervision, or establish the specific benefit of introducing this mechanism into iterative diffusion generation.
  2. The novelty of granularity-aware style modeling is insufficiently distinguished from prior work. Multi-level and multi-scale style representations have already been investigated by ArtAdapter and StyleShot. ArtAdapter explores different hierarchical style features and their effects through style mixing, while StyleShot extracts style information from multiple patch scales and evaluates their contributions. GASC instead operationalizes granularity primarily as token-shared versus token-varying conditioning. This is a potentially useful distinction, but the paper needs to explain more precisely what capability this formulation adds beyond existing multi-level representations and selective conditioning.
  3. The central allocation study does not isolate granularity–interface alignment. In Appendix C, Δ\Delta-S includes CLS and pools features before nonlinear mapping, whereas Δ\Delta-V excludes CLS and applies the mapping token-wise. Furthermore, Δ\Delta-V uses reference-grid features without content-conditioned retrieval, while BB-V uses C²R-derived carriers. Thus, the alternatives differ in feature construction and content alignment in addition to conditioning granularity. Although the appendix acknowledges some of these differences, Table 2 cannot conclusively attribute the improvement to the proposed alignment principle.
  4. The proposed functional relationship between interfaces and visual style granularity remains insufficiently validated. Equation (1) shows that Δ\Delta affects both state transition and input discretization, while information written through BB can propagate to subsequent positions. Their different computational roles do not directly establish a preference for global appearance versus local stylistic details. Moreover, shared and token-varying representations do not necessarily correspond to coarse and fine visual attributes. Table 2 supports S/V on FID and ArtFID, but S/V has the highest LPIPS and worse CFSD than S/S and both single-route variants. Although this trade-off is acknowledged, no comparison at matched stylization strength or direct evaluation of local stylistic fidelity is provided.
  5. HOD’s interpretation as suppressing content while preserving style is supported only indirectly. Removing leading principal components eliminates high-variance directions, but these directions are not necessarily specific to content. The analysis on 155 WikiArt images reports higher descriptive content association for the selected components, but does not demonstrate that their removal preserves stylistic information. Table 3(a) also lacks an independently trained variant that removes only HOD while retaining C²R. The fixed-checkpoint sensitivity study in Table 5 does not resolve this issue, since changing residualization at inference also changes the feature distribution relative to training.
  6. The component ablations do not sufficiently identify the sources of improvement. Equation (7) combines retrieved carriers with raw reference features at corresponding grid positions, even though those positions have no guaranteed semantic correspondence to the content image. Removing the entire C²R pathway does not distinguish the contribution of content-conditioned retrieval from that of the raw-feature branch or learned fusion. Separate retrieval-only, raw-feature-only, and fusion controls are needed to establish the proposed mechanism. Similarly, the Content Stream and Content-to-CC variants in Table 3(b) are explicitly not capacity-matched. Their difference therefore cannot be attributed solely to the injection location, particularly given the small CFSD improvement from 0.265 to 0.264.
Questions for authors

See weakness.

Reviewer_27rKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies reference-based arbitrary image style transfer using a pretrained diffusion–Mamba denoiser. Its central principle is granularity–interface alignment: an image-level, token-shared style descriptor conditions the state-transition pathway, while token-varying local style carriers condition the input-write pathway. The method also introduces content-suppressed style descriptors, content-conditioned carrier retrieval, and a spatially aligned content stream. Validation includes comparisons with conventional and diffusion-based style-transfer methods, component ablations, and an independently retrained route-allocation study.

Critical strengths

The central architectural hypothesis is tested directly rather than presented only as intuition. The route-allocation study shows that assigning shared global information and varying local information to different pathways consistently affects performance, with the proposed assignment producing the strongest distribution-level results. This provides meaningful support for the paper’s main conceptual contribution.

The method performs strongly on FID, ArtFID, and content-structure evaluation against a reasonably broad set of baselines. The component analyses further help connect these improvements to the proposed descriptor construction and carrier retrieval, strengthening the case that the gains arise from the intended design rather than an isolated engineering trick.

Critical weaknesses

The perceptual evaluation is limited in scale and lacks a human preference study, while the construction of the FID reference distribution is insufficiently specified. This is consequential because the content-preservation evidence is mixed: the proposed model performs substantially worse than the strongest content-oriented baseline on LPIPS, and some component or alternative-allocation configurations achieve better content-structure scores. The results therefore do not yet establish a uniformly better style–content trade-off.

The allocation experiment does not completely isolate conditioning granularity. Pooling, CLS-token treatment, and the placement of the nonlinear conditioning map differ between configurations. The content-path comparison also has capacity differences and lacks a straightforward no-content-conditioning baseline. These confounds weaken the causal attribution to granularity–interface alignment.

Questions for authors

Are the evaluation images strictly disjoint from images or derived targets in the training data, and exactly how is the reference distribution for FID constructed?

Can you provide a larger human evaluation, with uncertainty estimates, that separately measures style fidelity and content preservation against the strongest baselines?

Can you provide allocation controls with identical pooling, CLS-token treatment, and conditioning-map placement, together with a no-content-stream baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.