acceptodds
Under review as a conference paper at ICLR 2027

AlphaRAE: Joint Color and Opacity Modeling with Representation Autoencoders

Abstract

Transparent images require a representation that preserves foreground color and opacity even when a pretrained visual encoder observes only RGB composites. We introduce AlphaRAE, which augments frozen visual features with a spatial color–opacity code and decodes both through a shared RGBA interface. A conditional adapter supplies this interface from a frozen RGB diffusion transformer. At 1024 pixels, removing spatial detail costs 2.91 dB PSNR, while an alpha-only detail input worsens rendered and direct-opacity fidelity at the reported endpoint. On 434 locally reserved images, the high-resolution model reaches 38.83 dB PSNR and 0.01463 LPIPS, compared with 38.52 dB and 0.02123 for AlphaVAE. The 256-pixel comparison remains mixed, although both configurations reduce direct alpha error on local and external images. At matched latent and trainable budgets, the joint model exceeds an independent-alpha control by 0.60 ± 0.12 dB PSNR across three seeds. Same-RGB generation and recovery studies show that the shared decoder can also receive predicted joint latents; a 53-rater study favors its output over same-alpha color refinement on 64 fixed conditions. These results separate codec fidelity from the ability of a generator to supply its latent.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation autoencoders, RGBA reconstruction, transparent images

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2gYZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a method of generating RGB-Alpha images from a pre-trained RGB diffusion transformer model without having to modify the DiT. To achieve this, the authors train both a latent code modifier and an RGB-A decoder that expects the modified latent vector. The latent code modifier uses the encoding from the pre-trained DiT to predict the latent vector corrections necessary to generate a proper alpha-channel using their RGB-A decoder.

To train the RGB-A decoder and build the corrected latent space, the authors use a pre-trained ViT semantic encoder and a custom CNN to capture detail features. They then train the encoder/decoder pathway as an auto-encoder that takes in an RGB-A image and outputs the same image. However, the ViT semantic encoder sees only an RGB version of the original RGB-A image when building the semantic features. The CNN detail features are built from both the foreground and a mask and are combined with the semantic features to build the joint latent space.

During generation, the DiT generator builds a class-guided image and the adapter is trained to modify the latent representation, adding additional predicted alpha-channel information, which can then be provided to the trained decoder.

The primary contribution is the concept of how to train a correction model that can enhance the latent representation from the DiT to enable the trained RGB-A decoder to build an accurate alpha map. This concept could conceivably be used in other applications that want to make use of large scale models pre-trained on RGB images but that need additional outputs, such as depth, opacity, or other scene properties.

Critical strengths

The central concept, learning to modify an RGB generated latent vector in order to facilitate an RGB-A decoder, is interesting. While not motivated in the paper, this enables a large-scale diffusion model trained on RGB information to generate class-guided images that will include an alpha-channel. This means a pre-trained DiT can be used to generate RGBA assets, which are potentially useful for many tasks. (However, the paper does not clearly motivate the work with any real-world application examples, so this is my understanding of its potential.)

The quantitative results also seem to show that the author's design works better than comparative models, especially at generating the estimated alpha channel information. In the selected generated images, there is qualitative improvement in the jellyfish image, and a user study supports the new method.

The particular design choices are incremental, with the key insight being the use of a spatial detail channel to help build the teacher network that creates the modified latent representation. Note that the paper never really explains that one network is the teacher of the other.

Critical weaknesses

My primary concern is the writing. Normally, I would not consider writing a critical weakness. However, the writing in this case is so obtuse and unclear it affects the reader's ability to understand the work. I include a number of examples below to be specific in my concerns.

A minor concern is whether there was IRB approval for the user study. This is not stated in the paper or the ethics statement.

Given the lack of references in some of the critical design paragraphs, it is difficult to evaluate the novelty of the work.

Figure 1 is the critical figure in the paper, but it should include some of the vocabulary used there, such as K7 and K23, and the 'Adapter' box is not well-defined. I would suggest a breakout box showing what is in the 'Adapter'.

Writing concerns

Abstract: the first two sentence attempt to motivate the problem: encoding images with an alpha channel using an RGB-trained encoder. The third sentence uses the words conditional adapter and frozen RGB diffusion transformer, but it's not clear from that sentence what exactly that means in this context.

The remaining sentences are very difficult to parse. Sentence four immediately talks about removing spatial detail costing PSNR, but what is removing the spatial detail? How does that relate to encoding RGBA images? What does "alpha-only detail input' mean in this context? What 'high-resolution model' is being referenced in the following sentence that reaches 38.83 dB PSNR? Is that the proposed model or an existing model? What does high-resolution mean in this case? Is that 1024, or something larger? Why does the abstract talk about 'Same-RGB generation' when the paper is about encoding RGBA images?

In short, I don't feel like anyone reading the abstract is going to get enough of a sense of the paper for it to be useful. While this is primarily a writing issue, it is an important enough issue to put it on the edge of being a major weakness in the paper.

Line 047: 'Strong reconstruction alone does not establish that the pretrained RGB distribution is compatible with the new decoder.'

  • this sentence lacks context
    • this is the first mention of a new decoder (assume this is learning to generate images from the joint RGBA latent code)
    • why doesn't strong reconstruction establish compatibility? Strong reconstruction by what?

Line 051: 'Its detail code is packed onto the semantic token grid'

  • detail code of the compact spatial encoder? What is detail code?
  • what is the semantic token grid?
  • these things seem to be described in Figure 1, which is not referenced in this paragraph. I'd recommend putting figure 1 earlier and referencing it as well as making the description more clear.
  • overall this is one of many examples of forward referencing without context or explanation.

Line 054: "a conditional adapter predicts the complete joint code from the pretrained RGB generator’s output features and pixels."

  • However, figure 1 states that the conditional adapter predicts correction and detail codes which combine with the native RGB latent. That suggests that the conditional adapter does not predict the complete joint code, but instead predicts a modifier to the RGB DiT latent code, the combination of which is the final joint code passed to the RGBA decoder.

line 056: What is 'training-only pseudo opacity'? and what does it mean that they provide complementary conditions? Conditions of what? Conditions during when? Training?

Lines 057-058 also comes back to the motivation. Figure 1 suggests that AlphaRAE can be used for two purposes: (1) encoding an RGBA image into a latent space and then decoding it, and (2) generating an RGBA image from scratch using a pre-trained RGB generator model. Of the two applications, #2 seems useful for creating AI-generated assets that can be easily composited. The purpose of #1 beyond training the RGBA decoder is unclear. The boldfaced caption of figure 1 seems to suggest both are useful on their own.

This also ties back to the abstract. The problem the author's seem to be solving is to enable AI generation of RGBA assets using a pre-trained RGB encoder. But that isn't stated clearly in the abstract or the introduction.

Line 060: rather than say 434 local reserved images, it would be more clear to say a 434 image test set? This is part of a larger confusion where the AIM data set appears to have some images that are in another (local) data set. But it's not clear what 'local' means in this context. There are clearly some images in both the AIM and the 'local' data set, and results are shown for the non-overlapping local and AIM subsets. Why choose a test set that contains images in another data set?

line 062: 'Direct alpha evaluation nevertheless favors both configurations' does 'configuration' refer to K7 and K23, or to the two resolution versions of AlphaVAE? Again, it's unclear from the text which method is better.

Please state the contributions clearly at the end of the introduction.

  1. 'explicit RGBA interface' what does that actually mean?
  2. A study of where transparent information is useful (good)
  3. A latent modification to retain spatial detail

The rest of the 'contributions' paragraph is a summary of the results, which confuses the purpose of the paragraph.

line 107: 'Our codec uses real RGBA labels'

  • class conditioned generation does not have alpha labels
  • during training real alpha labels appear to be available
  • what does it mean that 'their codec uses real RGBA labels'?
    • real labels are used during training to learn how to adapt the RGB DiT outputs, but how is that 'codec uses real RGBA labels'? The inference system has to learn to build the RGBA labels from the class conditioned generated latent.

Line 148: b is not defined in the final equation of the paragraph. This appears to be an arbitrary background color.

line 150: The sentence "The white composite cannot identify opacity" is strange wording. A white composite image can't identify anything. Presumably it's some model or operator that would be doing the identification. 'A white composite image cannot uniquely represent opacity:' would be more clear and correct.

Figure 1 needs to be referenced and used in section 3.1. However, it is not referenced anywhere in the document.

Line 153: 'RAEv2 supplies the multi-layer semantic encoder and RGB decoder'

  • RAEv2 is listed as prior work, but is not referenced in figure 1 or elsewhere in the text. Is RAEv2 the RGB DiT in figure 1? If RAEv2 is supplying an RGB decoder, the prior paragraph says the reconstruction generates sRGB composites, and figure 1 just indicates an RGB ViT + readout in the box listed as RGBA decoder.

line 154: 'the detail stream follows the residual-code pattern of prior autoencoders'. But there is no citation or reference to prior work. What prior autoencoders are being referenced here?

line 157: Semantic and spatial encoding

'The frozen encoder Ek accepts linear RGB...' In figure 1 this is listed as DINOv3, assuming this is the top path of the RGBA reconstruction track. But the remainder of the paragraph talks about RAEv2 and the K7 and K23 models. Those are not shown in figure 1. The paragraph starting with line 166 describes the detail channel, which would be the lower path of the reconstruction track. What role does RAEv2 play in the selection of features from DINOv3?

This is explained a little more clearly in the appendix which connects DINOv3 with K7 and K23. But the paper shouldn't rely on the appendix for understanding to core concepts.

Line 213: if the weights don't depend on the observed magnitude of each loss, how were they selected?

Line 213: 'All training losses operate on continuous images; JPEG is confined to the matched evaluation protocol.'

This sentence is another use of a semi-colon and missing explanations that just makes things more difficult to read than necessary.

The phrase 'matched evaluation protocol' is not used anywhere else in the paper, and the sentence does not reference any other section or experiment in the paper, so what does it actually mean?

Also, images are not continuous. They can be compressed or uncompressed, and they can have different bit depths, but they are not continuous.

line 226: real-TRAIN is not defined. Line 948 in the appendix suggests that TRAIN is a training set, and VAL, and TEST are validation and test sets. But even that paragraph in the appendix is confusing. Half the data is excluded? What is this data used to train, given that line 264 talks about a corpus of 8k assets also divided into train/test/validation sets. It seems like TRAIN is actually used for testing alpha reconstruction in Table 2.

line 305: another sentence with a semi-colon. The second part of it 'active codec parameters are X versus Y at 1024.' The assumption is that the X refers to the AlphaRAE and Y refers to AlphaVAE, meaning the proposed system is using 9x as many active codec parameters. However, the following sentence seems to imply the two methods have roughly the same timing and memory usage? How?

Table 1 is not referenced directly in the text. It should be referenced somewhere in the Metrics paragraph (lines 310 to 316) and in the first paragraph of section 4.2.

Table two indicates L1 numbers, but no units. The 95% difference intervals are multiplied by 1000 (another semi-colon). I assume the range is [0, 1], which puts the differences at ~.7%, and the actual errors at .4% to 1.1%. This should be explicit in the table.

Lines 378-379: K23 reduces Alpha L1 on soft interiors compared to what? Compared to itself, compared to K7, or compared to AlphaVAE? This is not specified.

The conclusion just repeats sentences from the results section without making any conclusions or clearly identifying the contributions of the paper.

Questions for authors

What is the purpose of designing this network? What is the problem you are trying to solve?

The use of the two test sets in table 2 is not well explained. What is the Local TEST, and why does it overlap so much with the External AIM test set?

What are the primary contributions of the paper? What design aspects could be used for other applications?

How would you re-write the paper (it needs a lot of work) if it were accepted?

5 comments
AuthorsOpenReview ↗

1/3: Purpose, contributions, and concrete manuscript revisions

Thank you for the detailed reading and concrete examples. The original presentation compressed several explanations into short statements and left important terms undefined. We have rewritten the narrative across the abstract, method, experiments, discussion, and conclusion. We address the purpose and contribution here, the representation and training in Part 2, and the data, evaluation, and ethics in Part 3.

What problem does AlphaRAE solve? We want to reuse pretrained RGB representations and generators to create RGBA assets whose foreground color and opacity remain useful on new backgrounds, for image editing, graphic design, and composition. A white-background RGB composite cannot uniquely determine the underlying foreground and opacity. The codec preserves this additional information, and the adapter supplies the codec's inputs from an existing RGB generator.

Why evaluate reconstruction as well as generation? Reconstruction tests whether a representation preserves an existing asset's foreground color and opacity when decoded and composited onto different backgrounds. The codec also defines the latent targets and decoder used for adapter training. At generation, no target RGBA asset exists: the RGB generator supplies its native semantic latent and RGB pixels, but the RGBA decoder also needs the added detail component. Good reconstruction from encoded RGBA inputs therefore does not establish compatibility with generated RGB inputs. We evaluate these two capabilities separately.

The three contributions are now stated explicitly.

  1. An RGBA representation combining a frozen RGB semantic encoder with explicit opacity injection, a learned joint color-opacity detail code, and a shared spatial readout.
  2. A residual adaptation mechanism that predicts the missing detail and a semantic correction from a frozen RGB generator's outputs, allowing the same RGBA decoder to serve reconstruction and generation.
  3. An information-path study and evaluation that separate composite fidelity, opacity fidelity, and RGB-to-RGBA adaptation.

The reusable design principle is to preserve pretrained RGB semantic features, add detail for the target property, and learn a residual mapping from generated RGB inputs. We demonstrate this principle for opacity, as now stated in the conclusion.

How has the writing changed? We revised the surrounding explanations as well as the phrases you identified:

  • Abstract and introduction: the goal is stated directly as reusing a pretrained RGB generator for RGBA assets. The abstract explains both training stages, reports 67% lower reconstruction Alpha L1 than AlphaVAE on the 434 test assets at 1024 pixels, and states that adaptation follows completed RGB sampling. It no longer introduces K23 or ablation labels without definitions. The introduction ends with the three contributions above.
  • Section 3: the opening distinguishes DINOv3, RAEv2, and the RGB DiT. Section 3.1 gives the K7/K23 layer indices, resolutions, and RGB initializations in the main text. Section 3.3 first explains RGB sampling and adapter prediction, then the training pairs and supervision. “Native RGB latent” is defined as the latent supplied directly by the generator.
  • Section 4: comparisons now name the models and the meaning of each difference. Section 4.5 defines B1/B2 before using them. The Figure 3 caption now says “RGBA conversion of the same generated RGB input,” replacing the unclear “same-RGB” label.
  • Discussion and conclusion: these now explain what the codec and adaptation studies establish, the composite-quality/alpha-error tradeoff, and the dependence of our main generation route on RGB structure. The conclusion explicitly states the reusable design principle above, demonstrated here for opacity, and names higher-resolution generation and direct joint denoising as further directions.

Figure 1 is also referenced when the method is introduced and in Section 3. The main text now explains the components and key configurations directly, with the appendices supplying extended protocols and analyses.

AuthorsOpenReview ↗

2/3: Representation, two-stage training, and adapter supervision

DINOv3, RAEv2, and the RGB DiT have different roles. DINOv3 is the frozen visual backbone. RAEv2 supplies the multi-layer feature aggregation and pretrained RGB decoder. The RGB DiT is a separate generative model that samples RGB latents. We now name and cite the inherited components in Section 3.

K7 aggregates layers 11,13,15,17,19,21,23 and K23 aggregates layers 1–23, using zero-based indices. Both have 1024 semantic channels. RAEv2 normalizes and averages the selected features, adding the final selected layer's spatial mean. K7 is evaluated at 256 pixels from the official ImageNet RGB initialization. K23 is evaluated at 1024 pixels from RGB weights adapted at that resolution. These configurations are now specified in Section 3.1. The encoder wrapper accepts linear RGB, converts it to sRGB, and then applies DINOv3.

What is the detail code and how is it packed? A learned CNN receives four channels: premultiplied sRGB color and alpha. Its 16-channel output at stride eight is rearranged by factor-two pixel unshuffle into 64 channels at stride sixteen. This aligns it with the semantic grid of size H/16×W/16H/16 \times W/16. Concatenating the 1024 semantic and 64 detail channels gives the 1088-channel decoder input. The detail code carries color and opacity information; it is not an alpha mask attached after RGB decoding. Semantic opacity injection separately adds pixel-unshuffled (1−a)(1-a) with fixed scale 0.1 after DINO encoding.

The two training stages and supervision are now separated.

  • Codec training: real RGBA assets supply foreground color and alpha. We train the detail encoder, the entire RGB decoder, and the spatial readout, keeping DINOv3 frozen. The RGB decoder and shared readout reconstruct color and opacity jointly.
  • Adapter training: both the trained codec and RGB DiT are frozen. Only the adapter is updated. The frozen RGBA encoder supplies the joint-latent targets.

Real adapter examples use white- or natural-background composites of training RGBA assets, paired with their known RGBA targets. Generated examples instead use native DiT latents and decoded RGB. The real examples provide genuine opacity supervision, while the generated pairs expose the adapter to generator outputs.

Which model supplies which training target? The frozen RGBA encoder constructs joint-latent targets. BiRefNet separately supplies pseudo alpha for generated RGB training pairs; this is the “pseudo-label teacher” in Figure 1. At class-conditioned inference, neither target encoding nor BiRefNet is needed. The RGB DiT supplies the latent and image, the adapter predicts correction and detail, and the frozen RGBA decoder remains in the output path.

Your interpretation of the residual output was correct. We replaced the ambiguous “predicts the complete joint code” wording. For RGB latent zrz_r and image iri_r, the adapter predicts (δs,δd)(\delta_s,\delta_d) and constructs

z~=[zr+σrδs,;μd+σdδd].\tilde z=[z_r+\sigma_r\delta_s,\\;\mu_d+\sigma_d\delta_d].

Here σr\sigma_r, μd\mu_d, and σd\sigma_d are fixed normalization statistics. This is Equation 9 in the revision. The adapter receives no target alpha; at generation, class and noise drive the RGB DiT, followed by the adapter and frozen RGBA decoder. BiRefNet is not used at inference.

Figure 1 now identifies K7/K23, the variables and frozen/updated components, and the correction/detail composition. Its expanded adapter block lists the eight backbone blocks, two prediction-head blocks, and spatial refiner. DDT means Decoupled Diffusion Transformer, separating feature encoding from output prediction. Figure 1's caption and Section 3.3 now define and cite this design and distinguish its RAE/RAEv2 implementation. Real paired thumbnails show K23 codec inputs/reconstructions and K7 native RGB/adapter outputs, with checkerboards indicating transparency. Section 3.3 and Appendix D give the architecture and training details.

AuthorsOpenReview ↗

3/3: Data, evaluation definitions, measured cost, and ethics

Local TRAIN/VAL/TEST and the external set. The local corpus contains 8,635 RGBA assets. “Real-TRAIN” means its 7,779 training assets, used for codec training and real supervision of the adapter. The 422 validation assets serve development/model selection; the 434-image reconstruction test is excluded from fine-tuning. The 4,499 generated pseudo examples form a separate adapter-training pool.

The separately acquired external release is AlphaVAE's AlphaTest. It has 902 images: 500 from AIM-500 (Automatic Image Matting-500), introduced by Li, Zhang, and Tao, IJCAI 2021, and 402 Internet images. Exact-file screening against all local splits removes one AIM and 401 Internet matches, leaving AIM499 plus one Internet image. The latter passes the same exclusion rule and remains in the external evaluation, outside the AIM-only means. These exclusions concern the external evaluation pool, not half of the local training corpus. We also disclose that later aggregate RGB-conditioned recovery scores on the local test informed adapter development. Section 4.1 now defines and cites these sources, and Table 1 summarizes their roles.

Definitions and evaluation. We replaced the ambiguous expressions you identified:

  • Background bb is a constant sRGB color broadcast across the image. Different foreground/alpha pairs can produce the same white composite, so that composite cannot uniquely determine opacity.
  • “Continuous images” now means floating-point pixel values before byte quantization or JPEG. The matched codec protocol applies the same preprocessing, nine backgrounds, JPEG procedure, and metrics to all compared codecs.
  • Alpha lies in [0,1][0,1]. Alpha L1 is dimensionless; both means and difference intervals marked ×1000\times1000 are scaled by 1000.
  • “Both configurations” means AlphaRAE K7-256 and K23-1024, each compared with its resolution-matched AlphaVAE reference. The soft-interior result compares K23 with AlphaVAE at 1024: on 58 GT-defined external soft-interior images, unscaled regional Alpha L1 falls from AlphaVAE's 0.04026 to K23's 0.02627.
  • The original reconstruction Table 1 is now Table 2 because of the added data-role table. The result tables are explicitly referenced in the evaluation/reconstruction text; direct-alpha definitions and units accompany Table 3.

How were the loss weights selected? The coefficients 1000/200/50/80/6 were chosen empirically in preliminary runs, informed by existing reconstruction objectives. The final recipe remained fixed throughout the reported formal 50k-update runs. Section 3.2 now states this selection process and the floating-point training convention.

Why do parameter count and measured cost differ? Parameter count measures stored weights, while latency depends on the executed operations and feature resolutions. Peak allocation also includes memory used during execution. At 1024 on one A800 in FP32 with batch one, we measure:

MeasurementAlphaRAEAlphaVAE
Codec parameters, including frozen encoder723.5M83.8M
Complete-path latency (ms)625.3658.3
Codec-only latency (ms)588.7622.7
Allocation after loading (GiB)2.70480.3218
Additional runtime allocation (GiB)0.78563.0176
Peak allocation (GiB)3.49043.3394

AlphaVAE's larger additional allocation offsets its smaller weight footprint at the peak. Complete-path timing includes preprocessing, device transfers, encode/decode, and output conversion, excluding file I/O, loading, and metrics. Section 4.1 explains these distinctions, and Appendix A gives the protocol and breakdown.

Human-study review and consent. We did not seek institutional IRB review or an exemption determination for this questionnaire. The revised Ethics Statement explicitly reports this status and the procedure. The 53 student volunteers completed an anonymous, voluntary online comparison of generated-image pairs. Before starting, they were informed of the study's purpose and research use of their responses. No names, student IDs, email addresses, or other personal information were collected. Small random-amount red envelopes were distributed anonymously through a messaging app as appreciation, without linking recipients to responses. Only aggregate preferences are reported.

Reviewer_3nk3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

AlphaRAE aims to extend an existing RGB representation autoencoder (RAE) into an RGBA codec. The authors freeze DINOv3, inject alpha information into features extracted from white-background RGB composites, and use a CNN to encode premultiplied color and alpha into an additional 64-channel detail code. Together, these form a 1088-channel latent, from which the trained decoder reconstructs RGB and alpha. For generation, the method retains an existing K7 RGB DiT. After RGB denoising is complete, a separately trained adapter predicts semantic corrections and detail codes from the RGB image and its latent, which are then decoded into RGBA. The main experiments focus on RGBA reconstruction, while generation is evaluated only with K7 at 256 resolution.

Critical strengths
  1. Extending existing RAEs to RGBA is a practically meaningful goal. Existing visual representations and generative models mainly handle RGB, so exploring how to reuse these capabilities for transparent assets is a worthwhile research direction.

  2. I think the implementation of AlphaRAE makes sense overall. Since features from a white-background RGB composite cannot fully preserve the original asset's opacity, explicitly encoding additional alpha information is a natural choice. Having the detail code carry both color and alpha and decoding them jointly is also intuitively reasonable, and the authors provide relevant ablations to support this design.

Critical weaknesses
  1. I do not find the current adapter setup fully convincing as a validation of AlphaRAE's generative capabilities. If I understand correctly, the RGB DiT first completes denoising, and the adapter then predicts the codes needed by the RGBA decoder from the clean RGB latent and the RGB image. Functionally, this seems closer to matting and foreground-color recovery. It does modify RGB rather than simply predict an alpha map, but the experiment mainly shows that it can convert existing RGB outputs into RGBA. I would still like to see denoising-based generation directly in the RGBA latent space to better understand how well this representation works for generation.

  2. I am not yet convinced that the amount of RGBA training data is sufficient. There are only 7,779 RGBA assets used for training. Although the method benefits from pretrained models and additional RGB data, I think more evidence is needed to show that this is enough to obtain a useful RGBA codec that generalizes reliably. I also wonder whether the limited amount of RGBA data is one reason the authors did not conduct direct RGBA latent generation experiments. If so, I would appreciate a clearer discussion of this limitation.

  3. The paper is somewhat difficult to follow, and the connection between the method and experiments is not easy to grasp. Even with some familiarity with RAEs and generative models, I found myself having to read back and forth.

Questions for authors
  1. Could you add an experiment that performs denoising-based generation directly in the RGBA latent space? I would like to understand how directly generating the joint latent compares with the current approach of first generating RGB and then converting it through an adapter.

  2. I would like to see a version trained with more RGBA data, or at least results across different training-set sizes. If the authors believe the current amount of data is already sufficient, please provide more direct evidence that reconstruction, generalization, and generation performance are not substantially limited by the dataset size.

4 comments
AuthorsOpenReview ↗

1/2: Direct RGBA-latent denoising and the generation-route comparison

Thank you for the experimental suggestion. We agree that converting completed RGB samples does not by itself show whether the joint RGBA latent can be generated by denoising. We added that experiment in Appendix M, using the same real/pseudo training pools as the fresh adapter reference.

Direct generation setup. Direct samples the full 1088-channel K7 latent from class and Gaussian noise, without a clean RGB image or latent as input. It trains rank-64 LoRA and expanded input/output projections in the pretrained RGB denoiser. The reference completes RGB sampling and then applies an adapter trained from random initialization, separate from the original 47.5k adapter.

Both routes use the same frozen 50k K7 codec, released RGB prior checkpoint, 7,779 real and 4,499 pseudo assets, 75/25 real/pseudo mixture, global batch 32, and an initial budget of 8k updates. Real assets without class labels use the null condition, while pseudo assets retain their ImageNet classes. The routes have different conditioning and objectives, with 61.61M/36.34M trainable parameters for Direct/adapter. Equal data and updates therefore do not imply equal computation.

Observed behavior and longer training. We evaluate 64 fixed conditions from eight classes and eight sampling seeds. All eight classes occur in pseudo training. Both routes use 100 Euler steps and the same guidance scales. Initial semantic noise is shared, but their RGB outputs are not constrained to match.

Direct produces recognizable subjects in some cases alongside distorted or overlapping structures, gray texture, and mottled alpha. We continued the same Direct run to 32k updates, evaluating its 16k and 32k checkpoints against the fixed adapter8k reference. Some contours improve and other cases fragment, with structural and opacity artifacts remaining at 32k. Figure 9 shows four fixed-index cases across budgets. We inspected all 64 conditions at each Direct checkpoint. The adapter also inherits RGB failures such as montages, cropped subjects, and cage bars.

At the initial 8k endpoints, mean full-generation latency on one A800 is 6.506 s for Direct and 3.510 s for RGB then adapter, excluding loading and PNG serialization. Appendix M reports these complete-route measurements and the continuation protocol.

Relation to matting and foreground-color recovery. The RGB-conditioned stage is related to matting. The adapter predicts a joint latent from which the codec decodes foreground color and alpha. Table 5 uses B1, BiRefNet alpha with foreground-color refinement, and B2, the same refiner with AlphaRAE's alpha. B2 and AlphaRAE share alpha, so their comparison isolates the foreground-color predictions. On this paired recovery task with known target RGBA, AlphaRAE lowers foreground RGB L1 from B2's 0.02418 to 0.02308. This is color-recovery evidence, separate from the direct-denoising test.

What the direct experiment establishes. The joint code can be sampled from class and noise at the available data scale, with the reported image artifacts. The 8k comparison measures complete pipelines with different objectives and conditioning. The 16k/32k checkpoints are a budget diagnostic from one Direct training seed, with a fixed adapter8k reference. These results do not establish equal-compute superiority, convergence, or opacity accuracy on generated images without target mattes.

AuthorsOpenReview ↗

2/2: Three-seed training-size study, generalization, and experiment roles

Training-set size and data sufficiency. We agree that useful reconstruction at one data scale does not establish that the dataset is sufficient. Appendix L/Table 24 now reports nine fresh K23 runs: nested sets of 1,944, 3,892, and 7,779 real RGBA assets, each trained with three seeds for 50k updates. All runs share the RGB initialization, representation, loss recipe, replay pool, and validation/evaluation identities. TEST434 assesses local reconstruction, while AIM499 assesses external opacity reconstruction after exact-file exclusion.

The following are means across three training seeds, with sample standard deviations:

  • 1,944 assets: PSNR 38.690±0.04738.690\pm0.047 dB; LPIPS 0.01478±0.000060.01478\pm0.00006; Alpha L1 1.714±0.0111.714\pm0.011 on TEST434 and 1.688±0.0091.688\pm0.009 on AIM499.
  • 3,892 assets: PSNR 38.721±0.05438.721\pm0.054 dB; LPIPS 0.01475±0.000120.01475\pm0.00012; Alpha L1 1.713±0.0141.713\pm0.014 on TEST434 and 1.700±0.0161.700\pm0.016 on AIM499.
  • 7,779 assets: PSNR 38.779±0.04938.779\pm0.049 dB; LPIPS 0.01458±0.000090.01458\pm0.00009; Alpha L1 1.742±0.0121.742\pm0.012 on TEST434 and 1.724±0.0181.724\pm0.018 on AIM499.

Alpha L1 values above are multiplied by 1000; the underlying opacity range is [0,1][0,1]. The standard deviations describe training randomness on the fixed nested subsets, not repeated subset selection.

What do the trends establish? These runs measure codec sensitivity within the tested data range. They do not establish data sufficiency for both reconstruction and generation. From 25% to full data, PSNR improves and LPIPS decreases in all three paired training seeds. The Full-minus-25% LPIPS differences are -0.000083, -0.000196, and -0.000303 for seeds 20261109, 20261110, and 20261111, now also listed in Appendix L. The paired mean PSNR gain is 0.089 dB; rFID means improve from 2.227 to 2.137. This provides a measured appearance benefit from additional real assets under the fixed protocol.

Opacity follows a different trend: mean Alpha L1 is slightly higher at full data on both local and external cohorts. Thus the measured result is a tradeoff between composite fidelity and opacity error over 1,944–7,779 assets under a fixed 50k-update protocol. Smaller sets are revisited more often. We interpret these curves within that range and budget, rather than as evidence of data saturation.

The size intervention varies real data during codec training and measures reconstruction and external opacity behavior. Appendix M instead holds the real/pseudo pools fixed while comparing generation routes. It shows that direct generation was tested at the available data scale, but does not determine how generation quality would change with a larger RGBA corpus or whether data scarcity caused the observed artifacts.

How the experiments connect to the method. Section 3 and Figure 1 explain codec learning followed by RGB-conditioned adaptation. Table 1 identifies the training and evaluation roles of each data source. Section 4.4 and Appendix L measure codec sensitivity to data size. Section 4.5 evaluates RGB-to-RGBA conversion, while Appendix M tests direct joint-latent generation. The discussion states the conclusions and limits of these distinct experiments.

Thank you for your reply. These experiments and the clarifications in the revised manuscript address my main concerns. I will raise my overall rating to 3.

Thank you for taking the time to review our additional experiments and revisions. Your suggestions on direct generation in the RGBA latent space and the effect of training-set size were particularly valuable. They helped us strengthen the empirical evaluation and clarify both what our results support and where their limitations lie.

We are glad that these additions have addressed your main concerns, and we sincerely appreciate your willingness to reflect this positive reassessment in the overall rating on OpenReview. Thank you again for the thoughtful and constructive discussion, which has helped us improve the paper.

Reviewer_GrgSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper extends RAEv2 to RGBA images. For encoding, the proposed AlphaRAE uses a frozen RAEv2 RGB encoder, then uses pixel unshuffle to process the alpha channel, and finally adds it to the RAEv2 RGB feature. There is also a lightweight spatial encoder for the alpha channel and premultiplied sRGB. These two features are then concatenated to form the final RGBA code (Eqs. (1)–(2)). For decoding, the RGBA decoder contains an RGB Transformer decoder and extra spatial modules, and is fully trainable. There is also a learned readout interpolation. For class-conditional generation, the authors train a separate conditional adapter connected to the frozen RGB DiT.

Critical strengths
  1. The authors extend RAE to RGBA and provide detailed discussions, such as where transparency information enters, with much more in the supplementary material, which might inspire future research.
  2. The proposed AlphaRAE outperforms existing RGBA VAEs in terms of reconstruction.
Critical weaknesses
  1. The authors propose a new RGBA VAE, AlphaRAE, that outperforms existing RGBA VAEs for image reconstruction. However, the number of latent channels in other RGBA VAEs is much smaller than that in AlphaRAE. As mentioned in Section 4.1, AlphaRAE stores 17x more latents than AlphaVAE. Considering this much larger bottleneck, it is not surprising that AlphaRAE achieves better reconstruction performance.

  2. It is hard to tell the difference between AlphaVAE and AlphaRAE in Figures 2,4,5. The authors may provide zoomed-in views to better show the differences.

  3. As far as I am concerned, the goal of an RGBA VAE is not simply to design an RGBA VAE, but to use this VAE to build a framework for text-to-RGBA image generation. This is also the goal of LayerDiffuse, AlphaVAE, and Qwen-Image-Layered. However, the authors only provide simple class-to-RGBA results in Sec. 4.5, which are interesting, but the results in Figure 3 are far from traditional RGBA generation results. I doubt whether AlphaRAE really outperforms AlphaVAE for text-to-RGBA image generation, which is more important for the generative AI community. I suggest that the authors train AlphaRAE for text-to-RGBA generation and provide classical cases such as hair, fire, glass cups, and bubbles with transparent backgrounds.

Questions for authors
  1. Although the implementation code is provided, the method description in Section 3 is not clear enough. It would be better to add more mathematical variables to Figure 1 so that the notation is aligned with the main text, such as E_K, D_theta, s_hat, f, G, y, epsilon, i_r, A_psi, and a. It would also be helpful if the authors could provide a more detailed framework in the supplementary material.
4 comments
AuthorsOpenReview ↗

1/2: Latent capacity, design evidence, and method clarity

Thank you for recognizing the value of extending RGB representations to transparent assets. We agree that better reconstruction with a larger latent bottleneck does not establish a better architecture at equal capacity. We address this comparison here, followed by the text-generation concern and visual comparisons in Part 2.

What the 17x difference means for Table 2. At a common image resolution, AlphaRAE stores 1088(H/16)(W/16)1088(H/16)(W/16) latent scalars and AlphaVAE stores 16(H/8)(W/8)16(H/8)(W/8), giving a factor of 17. Their RGB-pretraining histories also differ. We have not trained a capacity-matched AlphaVAE. Table 2 therefore measures the complete configurations at their existing interfaces, without isolating the effect of the RGBA design from capacity or RGB pretraining. Section 4.1 and Appendix A state these comparison conditions.

What the extension contributes within the RGB RAE interface. The RGB RAE already has 1024 semantic channels. AlphaRAE adds 64 detail channels, a 6.25% increase relative to that interface, and uses them with explicit semantic opacity injection and a joint spatial readout. The practical target is to make this existing RGB representation usable for RGBA and compatible with an existing RGB generator.

Within that interface, the alpha-only detail ablation in Table 4 retains the 1088-channel latent shape and the same nominal trainable parameter count, 420.364M, while replacing the CNN's premultiplied-color/alpha input with repeated alpha. At the single-seed 50k endpoint on 422 validation assets, Full and alpha-only detail have PSNR 38.488 and 37.955 dB. This comparison supports a benefit from including color in the detail input at fixed nominal width and parameter count. It answers an information-choice question within AlphaRAE. It does not remove the capacity and pretraining differences between AlphaRAE and AlphaVAE. Section 4.1 and the discussion now distinguish these two questions.

Figure 1 and the supplementary framework. Figure 1 now aligns its notation with Section 3, including EKE_K, DθD_\theta, s^\hat s, ff, GG, yy, ϵ\epsilon, iri_r, AψA_\psi, and aa. It identifies K7/K23 and their resolutions, frozen/updated modules, and the RGB latent/image inputs to the adapter. The expanded adapter block lists its backbone, prediction head, and spatial refiner, with correction/detail outputs combined with the native RGB latent before frozen RGBA decoding. DDT means Decoupled Diffusion Transformer; its backbone/head design and RAE/RAEv2 implementation are now defined and cited in the figure caption and Section 3.3. Real paired thumbnails show K23 codec inputs/reconstructions and K7 native RGB/adapter outputs. Section 3 gives the equations, Appendix A gives codec/training details, and Appendix D specifies the adapter architecture and supervision.

AuthorsOpenReview ↗

2/2: Text-conditioned materials and visible reconstruction differences

Scope of the generative claim. We agree that text-to-RGBA generation is a central downstream goal for an RGBA codec. Our study addresses a narrower question: whether a frozen RGB generator can supply the input an RGBA decoder needs. We evaluate class-conditioned RGB-to-RGBA generation with the K7 DiT (Section 4.5) and conversion of hair, fire, glass, and bubbles from a frozen text-to-RGB model (Appendix K). Direct denoising in the joint 1088-channel latent is tested as an alternative route (Appendix M). Appendix K shows a flame retained by AlphaRAE but removed by B1 (BiRefNet alpha with foreground-color refinement), alongside gray residuals in glass and bubbles. Appendix M produces recognizable subjects but retains structural and opacity artifacts through 32k updates. We have not trained a text-conditioned RGBA denoiser and do not claim superiority over AlphaVAE's complete text-to-RGBA pipeline. The abstract, Section 4.5, and discussion now describe the generative contribution as reuse of RGB priors. Training a text-conditioned RGBA denoiser on this representation is a natural next step.

Hair, fire, glass cups, and bubbles. Appendix K uses eight prompts per material and two fixed seeds, giving 64 conditions, with all outputs retained. The frozen SVG Pure-P model generates 1024-pixel RGB. We resize in sRGB to 256 pixels, convert to linear RGB, re-encode with K7, and apply the existing adapter and frozen RGBA codec. B1 and AlphaRAE receive identical RGB pixels.

The text enters only the frozen SVG model. The adapter, trained earlier on real RGBA pairs and K7 class-conditioned RGB outputs, is applied to re-encoded SVG images without further training. Text can affect the output through RGB, but these experiments do not establish independent text control over opacity.

Figure 8 shows the first prompt and first seed for each material. Hair detail is softened, B1 looks cleaner in the glass case, and bubbles retain uneven interior opacity and block-like gray texture. The demonstration is qualitative because the generated images have no reference matte. It reuses the 50k-update K7 codec and the adapter with 47.5k total updates across sequential stages, including its final 8k stage. The latter is separate from the fresh 8k comparison adapter in Appendix M.

Making Figures 2, 4, and 5 informative. Figure 2 now has matched detail insets. Figures 4 and 5 add identical-coordinate crops of RGB, alpha, and black/white composites. Crop positions use ground-truth structure, and alpha retains its original display range. The enlarged views expose boundary and opacity differences that full-image RGB panels obscure.

Dear Reviewer GrgS,

Thank you again for the review. In case it is useful before the discussion period closes, here is where your three points are addressed in the revision:

  • Latent capacity: we agree that Table 2 does not isolate an architectural advantage at equal capacity. We state this in Section 4.1 and the discussion, and we have not trained a capacity-matched AlphaVAE. Part 1/2 of our response also explains what the within-AlphaRAE controls do and do not show.
  • Figures 2, 4 and 5: they now include matched zoomed views with identical crop coordinates.
  • Text-to-RGBA: Appendix K shows hair, fire, glass and bubbles through a frozen text-to-RGB model, including failure cases, and Appendix M tests direct denoising of the joint latent. We have not trained a text-conditioned RGBA denoiser, and the revised text describes the generative contribution as reuse of RGB priors.

If any part of the response or the revised manuscript is still unclear, we would be glad to clarify it.

Dear Reviewer GrgS,

Thank you for taking the time to review our paper. Your feedback has helped us improve the manuscript.

As the author–reviewer discussion period draws to a close, we would appreciate it if you could review our rebuttal and let us know whether our responses address your concerns. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of author responses and revisions

We thank the reviewers and the Area Chair for their time. This note summarizes what changed in the revised manuscript and where each concern is answered.

Where to find the responses

  • Reviewer 2gYZ (writing, definitions, data, ethics): parts 1/3, 2/3 and 3/3 under their review.
  • Reviewer GrgS (latent capacity, figures, text-to-RGBA): parts 1/2 and 2/2 under their review.
  • Reviewer 3nk3 (direct RGBA-latent denoising, training-set size): parts 1/2 and 2/2 under their review. Reviewer 3nk3 has confirmed that the added experiments and clarifications address their main concerns, and we thank them for the follow-up.

What changed

  1. Writing and structure. The abstract, introduction, Section 3, discussion and conclusion were rewritten. The introduction now states the two questions (what to add to a frozen RGB representation, and how a pretrained RGB generator can supply it) and three contributions. Figure 1 uses the same notation as Section 3 and expands the adapter. New Table 1 lists the role of every data source. Evaluation definitions, loss selection, measured cost, and the human-study procedure are stated in the main text.
  2. New experiments. Appendix L trains nine K23 models on three nested dataset sizes with three seeds each. Appendix M compares direct denoising of the 1088-channel joint latent with the RGB-then-adapter route on 64 fixed conditions, continuing the direct run to 16k and 32k updates. Appendix K applies the frozen adapter to a frozen text-to-RGB model on hair, fire, glass, and bubbles. Figures 2, 4, and 5 now include matched zoomed views.

What the evidence does and does not support

  • Table 2 compares complete codec configurations at their native interfaces. AlphaRAE stores 17x more latent scalars than AlphaVAE, and we have not trained a capacity-matched AlphaVAE. The within-AlphaRAE controls test information placement, not capacity.
  • We have not trained a text-conditioned RGBA denoiser, and we do not claim superiority over AlphaVAE's complete text-to-RGBA pipeline. Our generative evidence concerns reusing frozen RGB priors. In the text experiment, text acts only through the RGB generator.
  • Direct joint-latent denoising produces recognizable subjects but keeps structural and opacity artifacts through 32k updates, so we do not claim it matches the RGB-then-adapter route.
  • The training-set study measures codec sensitivity over 1,944 to 7,779 assets. Composite fidelity improves slightly while mean Alpha L1 is slightly higher at full data. It does not show that the data is sufficient.
  • The human preference study did not undergo institutional IRB review or an exemption determination. The Ethics Statement reports its procedure and this status.

The revised PDF is 10 main-text pages plus appendices. We are glad to answer any further questions in the individual threads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.