acceptodds
Withdrawn from ICLR 2027

On Color Alignment in VAE Latent Spaces and Its Applications

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Variational autoencoders (VAEs) are a key part of modern text-to-image models, which generate images within their latent space. VAEs are known to disentangle the main factors of variation in the data, and color is known to be one of the most structured of these in natural images: decorrelating it yields one luminance axis and two opponent-color axes. Color should therefore be expected to emerge as a distinct factor in the VAE latent space. Yet how these latent spaces represent color remains largely unexplored. In this work, we show that the VAEs of text-to-image models share a color subspace aligned with brightness and opponent-colors. Through a linear approximation of the encoder and targeted latent steering, we find this subspace consistently across a broad range of VAEs, from SD1.5 to FLUX.2 and Z-Image. Building on this characterization, we propose three applications: ColorTuning, which achieves state-of-the-art in precise numerical color generation on the fine-grained CSS3/X11 system of GenColorBench, saturation control, to adjust the global chromatic intensity, and color transfer, to change the palette to match a reference.

Primary area:
generative models
Keywords:
Text-to-image generation, Computational color, Controllable generation, Natural image statistics, Latent steering

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_g6wWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper probes the color subspace of vae latents. The paper first shows that perturbing individual latent channels can consistently change the color of decoded images, although color and spatial information are only partially disentangled. It then analyzes a linear approximation of the VAE encoder and argues that the latent space contains opponent-color structures similar to luminance and chromatic opponent axes. Based on these observations, the paper further extracts a low-dimensional color basis by measuring how latent perturbations change the decoded colors in CIELAB space. This color basis is then used for three applications: numerical color precision, saturation control, and color transfer.

Critical strengths
  1. The probing and the analysis of the color subspace of vae latents is very interesting and insightful.
  2. The qualitative results look very good.
  3. I feel the saturation reduction might be very useful in distillation as distillation is likely to create over-saturated images. It might have a huge impact in the future.
Critical weaknesses
  1. I am really confused about the writing. In section 3.2, in the first half, it said that this paper will use Jacobian to do the analysis and its motivation to use Jacobian, but later it actually uses pseudoinverse. I want to point out they are not the same thing at all. In section 4, the application section, the paper uses a totally different method from section 3.2, the analysis section. I feel the relationship between section 3.1 and section 4 is larger, and section 3.2 seems completely unnecessary and irrelevant, especially when section 3.2 is put between section 3.1 and section 4.1.
  2. This paper lacks some important ablation. Please see Questions.
Questions for authors
  1. In section 3.2, the paper uses natural RGB image patches to do the analysis. I am wondering whether we can do an ablation on this analysis. For example, use black-white images and use random encoders.
  2. In section 4.2, can we directly use MLP to predict the target shift rather than via U in SVD?
  3. Figure 5 looks very nice, but it has only FLUX2 result. I am wondering if the qualitative performance of all those models related to the CIE chromatic diagrams in bottom of Figure 4?
Reviewer_dtqrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper discusses the representation of color within the latent space of the variational autoencoder of diffusion / flow matching (text-to-image) models. The authors visualize the impact of steering on channels associated with colors, verifying findings from previous work that color is (roughly) linearly encoded in the embedding space. They similarly approximate the linear projections of the encoder, finding that it also (roughly) linearly encodes color. They provide numerical results on the accuracy of color interventions, as well as qualitative results. They also provide qualitative results on color transfer and saturation adjustments.

Critical strengths

S1) The core problem is interesting. If the work had more clear takeaways that differentiate it from existing work, it has the potential to have interesting insights.

Critical weaknesses

At a high-level, I see two primary critical weaknesses of this work, that each repeat throughout the manuscript. For each, I include specific examples.

W1. Lack of and misrepresentation of novelty. The key contributions have very high overlap with existing work, and many of the claims of novelty stem either from under-representing existing works, or over-representing the work presented in this paper. Specifically:

W1a. Generalization to more VAEs. The first bullet-pointed contribution mentions showing a color subspace's existence in a variety of models (5), when in fact it has already been shown in a variety of models (SD in [1]; FLUX, SD3.5, FLUX.2, Qwen-Image in [2]). The discussion of which models have already been tested also reports [2] as only showing results for FLUX, when in fact, the paper includes 3 more models.

W1b. Misrepresentation of the Encoder Experiments. I find the analysis results to be misleading in terms of explanation of what they mean (section 3.2). First, it is claimed that the linear approximation of the VAE will show why color is readable from the latent channels, but the experiment is closer to showing how, aka the path from image to the latent. Furthermore, in the last sentence, it is indicated that this experiment should verify that the structure originates in the encoder, but I find it unclear why this must be verified, as the encoder is the only place it could have originated (as this is the only component between the image and the latent). If the authors see an alternative, this should be stated explicitly, as it is non-obvious, and it is otherwise unclear why this experiment provides new insights.

W1c. Overrepresentation of the Applications. The proposal of several applications is a claimed contribution, but quantitative results are only presented on numerical color precision, with minimal qualitative examples and no quantitative measurements of how accurately this method applies to color transfer or saturation control.

W1d. Unclear how Insights Differ from Previous Work. While the paper claims novelty from new insights, I find several of the insights to be fundamentally the same as what was shown in previous work. 1) The work claims that some fine-grained details are also encoded in the color channels (section 3.1), because the details change when Δz\Delta z is large. However, growing z infinitely will eventually push the latent out of distribution, so it is unclear if the artifacts are because other details are encoded on the channels, or the latent is simply OOD. [2] also claimed that modifications should be controlled. 2) [1] already indicated that color directions are close to channels, and the final finding here is also that they are not perfectly aligned is also the finding in [2].

W1e. Writing Overclaims Contributions. In addition to the details described above, the abstract reads like the key findings of other papers were discovered in this work (In this work, we show that the VAEs of text-to-image models share a color subspace aligned with brightness and opponent-colors. Through a linear approximation of the encoder and targeted latent steering, we find this subspace consistently across a broad range of VAEs). The title has a similar problem, implying claim over existing discoveries rather than what is novel to this work.

W1f. Overclaiming the Contribution on Real Images. The work claims that analysis on real images was not done before, and is instead a novelty of this work. However, both [1] and [2] verified the findings on real images, providing no true novelty from the real images used here.

W2. Insufficient comparisons. The results presented are missing many important pieces of information to properly assess if this method is better than SOTA color-modification methods, let alone how if fully compares to them. Specifically:

W2a. Missing Key SOTA Comparison. Although [2] was cited as a heavily related method, it is never compared quantitatively or qualitatively. This is especially problematic given the methodological overlap (see W1), with only small implementation details differentiating the two (how to find the directions, how far to go). There are also several key ways in which this work is disadvantageous compared to [2], including requiring additional training per-color, requiring an extra model (SAM), using a simplified color modification strategy which causes the loss of texture (see Figure 5, elephants), issues that were addressed in [2]. For these reasons, it is especially important that the method provided here shows advantages, to justify the disadvantages.

W2b. Other Missing SOTA Methods. Other key methods are missing as well, including ColorPeel [3] and best of N (a strong baseline, given that color modification methods are somewhat limited).

W2c. Missing baselines. The baseline compared to in this work is giving the model the hex code--this may over-exaggerate the problem in the base model, given that they understand natural language better than codes, and that the method presented here also modifies the prompt to the color name. It would be important to also compare to the out-of-the-box model with the color name.

W2d. Missing non-color comparison. It is important to also understand how much the method modifies non-color attributes, and if there are any other impacts on the images (like changes to diversity). Hence, it would be important to also compare other metrics like structural similarity (e.g. IoU, SSIM, LPIPS), diversity (e.g. pairwise DiNOv2 score), and image quality (e.g. Aesthetics score).

W2e. Weak results. While the method improves upon the currently compared-to methods on CSS3/X11, poor performance on ISCC-L3 shows a lack of generalization.

W2f. Inaccurate Linear Representations. As discussed in the last paragraph in section 3.2, the reliability of the linear approximations to the VAE relies on accuracy. However, Table 2 shows that they typically only explain about 75% of the energy within the network, leaving about 1/4 unaccounted for.

References:

[1] Color in coding in the latent space of stable diffusion models, Arias et al, Color and Imaging Conference Final Program'25.

[2] The latent color subspace, Pach et al, ICML'26.

[3] Colorpeel: Color prompt learning with diffusion models via color and shape disentanglement. Butt et al, ECCV'24.

Questions for authors

Q1. I find the motivations of Section 3.2 confusing, as well as the interpretation of the outputs--if I have misinterpreted it (see W1b), an explanation would be appreciated.

Q2. In addition, in Section 3.2, if the authors could describe for me the motivation behind confirming the color structure to originate in the encoder, it would be appreciated--specifically, where else do they see it could originate?

Reviewer_S25FOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how color is represented in the latent spaces of VAEs used in text-to-image models. The authors show that these latent spaces contain directions corresponding to luminance and opponent colors across several architectures, including SDXL, SD3, and FLUX. Based on this observation, they propose ColorTuning, which uses learned latent displacements to improve numerical color accuracy during generation. The method shows strong improvements on GenColorBench, particularly for CSS3/X11 colors.

Critical strengths
  1. The motivation is clear. Linking efficient coding and natural color statistics to the latent spaces that modern generators actually use is a nice angle.

  2. ColorTuning is practical. It needs no backbone retraining, it works on six backbones, and the CSS3/X11 gains are large.

Critical weaknesses
  1. Limited novelty over prior work - The paper discusses "Color encoding in Latent Space of Stable Diffusion Models" Arias et al. (2025) and "The Latent Color Subspace: Emergent Order in High-Dimensional Chaos" Pach et al. (2026), which have already identified structured color representations in the latent spaces of diffusion-model VAEs. In particular, Pach et al. also use a color subspace for controlling the colors of generated objects. I would like the authors to explain more clearly what new understanding is obtained beyond the existing observation that VAE latent spaces contain controllable color directions.

  2. The authors acknowledge that large latent shifts can distort image structure. However, the final evaluation mainly focuses on color accuracy. It is difficult to ascertain whether the improvements come at the expense of image quality or structural consistency. The current user study also evaluates only color matching and no aspect of structural consistency.

  3. While the improvements on CSS3/X11 are impressive, ColorTuning consistently underperforms NumColor on ISCC-L3 across all three comparable backbones. Also Table 1 shows ColorTuning scores lower than the base FLUX.2 (39.29 → 38.91). The paper does not adequately explain these discrepancies, which weakens the claims about numerical color control. The method requires intermediate decoding, SAM 3 segmentation, and additional latent interventions. Reporting the inference-time overhead would help assess its practical usefulness.

  4. Latent spatial resolution limitation in proposed ColorTuning - The quantitative evaluation mainly considers assigning a target color to a single object. It is unclear how well the method handles multiple objects with different target colors, particularly when objects are small, adjacent, or overlapping. Since SAM 3 masks are downsampled to the VAE's relatively low-resolution latent space, color perturbations may leak across object boundaries. Moreover, the paper does not clearly explain how different segmented regions are assigned colors from the reference palette.

Questions for authors

See weakness above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.