- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents a method of generating RGB-Alpha images from a pre-trained RGB diffusion transformer model without having to modify the DiT. To achieve this, the authors train both a latent code modifier and an RGB-A decoder that expects the modified latent vector. The latent code modifier uses the encoding from the pre-trained DiT to predict the latent vector corrections necessary to generate a proper alpha-channel using their RGB-A decoder.
To train the RGB-A decoder and build the corrected latent space, the authors use a pre-trained ViT semantic encoder and a custom CNN to capture detail features. They then train the encoder/decoder pathway as an auto-encoder that takes in an RGB-A image and outputs the same image. However, the ViT semantic encoder sees only an RGB version of the original RGB-A image when building the semantic features. The CNN detail features are built from both the foreground and a mask and are combined with the semantic features to build the joint latent space.
During generation, the DiT generator builds a class-guided image and the adapter is trained to modify the latent representation, adding additional predicted alpha-channel information, which can then be provided to the trained decoder.
The primary contribution is the concept of how to train a correction model that can enhance the latent representation from the DiT to enable the trained RGB-A decoder to build an accurate alpha map. This concept could conceivably be used in other applications that want to make use of large scale models pre-trained on RGB images but that need additional outputs, such as depth, opacity, or other scene properties.
Critical strengths
The central concept, learning to modify an RGB generated latent vector in order to facilitate an RGB-A decoder, is interesting. While not motivated in the paper, this enables a large-scale diffusion model trained on RGB information to generate class-guided images that will include an alpha-channel. This means a pre-trained DiT can be used to generate RGBA assets, which are potentially useful for many tasks. (However, the paper does not clearly motivate the work with any real-world application examples, so this is my understanding of its potential.)
The quantitative results also seem to show that the author's design works better than comparative models, especially at generating the estimated alpha channel information. In the selected generated images, there is qualitative improvement in the jellyfish image, and a user study supports the new method.
The particular design choices are incremental, with the key insight being the use of a spatial detail channel to help build the teacher network that creates the modified latent representation. Note that the paper never really explains that one network is the teacher of the other.
Critical weaknesses
My primary concern is the writing. Normally, I would not consider writing a critical weakness. However, the writing in this case is so obtuse and unclear it affects the reader's ability to understand the work. I include a number of examples below to be specific in my concerns.
A minor concern is whether there was IRB approval for the user study. This is not stated in the paper or the ethics statement.
Given the lack of references in some of the critical design paragraphs, it is difficult to evaluate the novelty of the work.
Figure 1 is the critical figure in the paper, but it should include some of the vocabulary used there, such as K7 and K23, and the 'Adapter' box is not well-defined. I would suggest a breakout box showing what is in the 'Adapter'.
Writing concerns
Abstract: the first two sentence attempt to motivate the problem: encoding images with an alpha channel using an RGB-trained encoder. The third sentence uses the words conditional adapter and frozen RGB diffusion transformer, but it's not clear from that sentence what exactly that means in this context.
The remaining sentences are very difficult to parse. Sentence four immediately talks about removing spatial detail costing PSNR, but what is removing the spatial detail? How does that relate to encoding RGBA images? What does "alpha-only detail input' mean in this context? What 'high-resolution model' is being referenced in the following sentence that reaches 38.83 dB PSNR? Is that the proposed model or an existing model? What does high-resolution mean in this case? Is that 1024, or something larger? Why does the abstract talk about 'Same-RGB generation' when the paper is about encoding RGBA images?
In short, I don't feel like anyone reading the abstract is going to get enough of a sense of the paper for it to be useful. While this is primarily a writing issue, it is an important enough issue to put it on the edge of being a major weakness in the paper.
Line 047: 'Strong reconstruction alone does not establish that the pretrained RGB distribution is compatible with the new decoder.'
- this sentence lacks context
- this is the first mention of a new decoder (assume this is learning to generate images from the joint RGBA latent code)
- why doesn't strong reconstruction establish compatibility? Strong reconstruction by what?
Line 051: 'Its detail code is packed onto the semantic token grid'
- detail code of the compact spatial encoder? What is detail code?
- what is the semantic token grid?
- these things seem to be described in Figure 1, which is not referenced in this paragraph. I'd recommend putting figure 1 earlier and referencing it as well as making the description more clear.
- overall this is one of many examples of forward referencing without context or explanation.
Line 054: "a conditional adapter predicts the complete joint code from the pretrained RGB generator’s output features and pixels."
- However, figure 1 states that the conditional adapter predicts correction and detail codes which combine with the native RGB latent. That suggests that the conditional adapter does not predict the complete joint code, but instead predicts a modifier to the RGB DiT latent code, the combination of which is the final joint code passed to the RGBA decoder.
line 056: What is 'training-only pseudo opacity'? and what does it mean that they provide complementary conditions? Conditions of what? Conditions during when? Training?
Lines 057-058 also comes back to the motivation. Figure 1 suggests that AlphaRAE can be used for two purposes: (1) encoding an RGBA image into a latent space and then decoding it, and (2) generating an RGBA image from scratch using a pre-trained RGB generator model. Of the two applications, #2 seems useful for creating AI-generated assets that can be easily composited. The purpose of #1 beyond training the RGBA decoder is unclear. The boldfaced caption of figure 1 seems to suggest both are useful on their own.
This also ties back to the abstract. The problem the author's seem to be solving is to enable AI generation of RGBA assets using a pre-trained RGB encoder. But that isn't stated clearly in the abstract or the introduction.
Line 060: rather than say 434 local reserved images, it would be more clear to say a 434 image test set? This is part of a larger confusion where the AIM data set appears to have some images that are in another (local) data set. But it's not clear what 'local' means in this context. There are clearly some images in both the AIM and the 'local' data set, and results are shown for the non-overlapping local and AIM subsets. Why choose a test set that contains images in another data set?
line 062: 'Direct alpha evaluation nevertheless favors both configurations' does 'configuration' refer to K7 and K23, or to the two resolution versions of AlphaVAE? Again, it's unclear from the text which method is better.
Please state the contributions clearly at the end of the introduction.
- 'explicit RGBA interface' what does that actually mean?
- A study of where transparent information is useful (good)
- A latent modification to retain spatial detail
The rest of the 'contributions' paragraph is a summary of the results, which confuses the purpose of the paragraph.
line 107: 'Our codec uses real RGBA labels'
- class conditioned generation does not have alpha labels
- during training real alpha labels appear to be available
- what does it mean that 'their codec uses real RGBA labels'?
- real labels are used during training to learn how to adapt the RGB DiT outputs, but how is that 'codec uses real RGBA labels'? The inference system has to learn to build the RGBA labels from the class conditioned generated latent.
Line 148: b is not defined in the final equation of the paragraph. This appears to be an arbitrary background color.
line 150: The sentence "The white composite cannot identify opacity" is strange wording. A white composite image can't identify anything. Presumably it's some model or operator that would be doing the identification. 'A white composite image cannot uniquely represent opacity:' would be more clear and correct.
Figure 1 needs to be referenced and used in section 3.1. However, it is not referenced anywhere in the document.
Line 153: 'RAEv2 supplies the multi-layer semantic encoder and RGB decoder'
- RAEv2 is listed as prior work, but is not referenced in figure 1 or elsewhere in the text. Is RAEv2 the RGB DiT in figure 1? If RAEv2 is supplying an RGB decoder, the prior paragraph says the reconstruction generates sRGB composites, and figure 1 just indicates an RGB ViT + readout in the box listed as RGBA decoder.
line 154: 'the detail stream follows the residual-code pattern of prior autoencoders'. But there is no citation or reference to prior work. What prior autoencoders are being referenced here?
line 157: Semantic and spatial encoding
'The frozen encoder Ek accepts linear RGB...' In figure 1 this is listed as DINOv3, assuming this is the top path of the RGBA reconstruction track. But the remainder of the paragraph talks about RAEv2 and the K7 and K23 models. Those are not shown in figure 1. The paragraph starting with line 166 describes the detail channel, which would be the lower path of the reconstruction track. What role does RAEv2 play in the selection of features from DINOv3?
This is explained a little more clearly in the appendix which connects DINOv3 with K7 and K23. But the paper shouldn't rely on the appendix for understanding to core concepts.
Line 213: if the weights don't depend on the observed magnitude of each loss, how were they selected?
Line 213: 'All training losses operate on continuous images; JPEG is confined to the matched evaluation protocol.'
This sentence is another use of a semi-colon and missing explanations that just makes things more difficult to read than necessary.
The phrase 'matched evaluation protocol' is not used anywhere else in the paper, and the sentence does not reference any other section or experiment in the paper, so what does it actually mean?
Also, images are not continuous. They can be compressed or uncompressed, and they can have different bit depths, but they are not continuous.
line 226: real-TRAIN is not defined. Line 948 in the appendix suggests that TRAIN is a training set, and VAL, and TEST are validation and test sets. But even that paragraph in the appendix is confusing. Half the data is excluded? What is this data used to train, given that line 264 talks about a corpus of 8k assets also divided into train/test/validation sets. It seems like TRAIN is actually used for testing alpha reconstruction in Table 2.
line 305: another sentence with a semi-colon. The second part of it 'active codec parameters are X versus Y at 1024.' The assumption is that the X refers to the AlphaRAE and Y refers to AlphaVAE, meaning the proposed system is using 9x as many active codec parameters. However, the following sentence seems to imply the two methods have roughly the same timing and memory usage? How?
Table 1 is not referenced directly in the text. It should be referenced somewhere in the Metrics paragraph (lines 310 to 316) and in the first paragraph of section 4.2.
Table two indicates L1 numbers, but no units. The 95% difference intervals are multiplied by 1000 (another semi-colon). I assume the range is [0, 1], which puts the differences at ~.7%, and the actual errors at .4% to 1.1%. This should be explicit in the table.
Lines 378-379: K23 reduces Alpha L1 on soft interiors compared to what? Compared to itself, compared to K7, or compared to AlphaVAE? This is not specified.
The conclusion just repeats sentences from the results section without making any conclusions or clearly identifying the contributions of the paper.
Questions for authors
What is the purpose of designing this network? What is the problem you are trying to solve?
The use of the two test sets in table 2 is not well explained. What is the Local TEST, and why does it overlap so much with the External AIM test set?
What are the primary contributions of the paper? What design aspects could be used for other applications?
How would you re-write the paper (it needs a lot of work) if it were accepted?
5 comments
2/3: Representation, two-stage training, and adapter supervision
DINOv3, RAEv2, and the RGB DiT have different roles. DINOv3 is the frozen visual backbone. RAEv2 supplies the multi-layer feature aggregation and pretrained RGB decoder. The RGB DiT is a separate generative model that samples RGB latents. We now name and cite the inherited components in Section 3.
K7 aggregates layers 11,13,15,17,19,21,23 and K23 aggregates layers 1–23, using zero-based indices. Both have 1024 semantic channels. RAEv2 normalizes and averages the selected features, adding the final selected layer's spatial mean. K7 is evaluated at 256 pixels from the official ImageNet RGB initialization. K23 is evaluated at 1024 pixels from RGB weights adapted at that resolution. These configurations are now specified in Section 3.1. The encoder wrapper accepts linear RGB, converts it to sRGB, and then applies DINOv3.
What is the detail code and how is it packed? A learned CNN receives four channels: premultiplied sRGB color and alpha. Its 16-channel output at stride eight is rearranged by factor-two pixel unshuffle into 64 channels at stride sixteen. This aligns it with the semantic grid of size . Concatenating the 1024 semantic and 64 detail channels gives the 1088-channel decoder input. The detail code carries color and opacity information; it is not an alpha mask attached after RGB decoding. Semantic opacity injection separately adds pixel-unshuffled with fixed scale 0.1 after DINO encoding.
The two training stages and supervision are now separated.
- Codec training: real RGBA assets supply foreground color and alpha. We train the detail encoder, the entire RGB decoder, and the spatial readout, keeping DINOv3 frozen. The RGB decoder and shared readout reconstruct color and opacity jointly.
- Adapter training: both the trained codec and RGB DiT are frozen. Only the adapter is updated. The frozen RGBA encoder supplies the joint-latent targets.
Real adapter examples use white- or natural-background composites of training RGBA assets, paired with their known RGBA targets. Generated examples instead use native DiT latents and decoded RGB. The real examples provide genuine opacity supervision, while the generated pairs expose the adapter to generator outputs.
Which model supplies which training target? The frozen RGBA encoder constructs joint-latent targets. BiRefNet separately supplies pseudo alpha for generated RGB training pairs; this is the “pseudo-label teacher” in Figure 1. At class-conditioned inference, neither target encoding nor BiRefNet is needed. The RGB DiT supplies the latent and image, the adapter predicts correction and detail, and the frozen RGBA decoder remains in the output path.
Your interpretation of the residual output was correct. We replaced the ambiguous “predicts the complete joint code” wording. For RGB latent and image , the adapter predicts and constructs
Here , , and are fixed normalization statistics. This is Equation 9 in the revision. The adapter receives no target alpha; at generation, class and noise drive the RGB DiT, followed by the adapter and frozen RGBA decoder. BiRefNet is not used at inference.
Figure 1 now identifies K7/K23, the variables and frozen/updated components, and the correction/detail composition. Its expanded adapter block lists the eight backbone blocks, two prediction-head blocks, and spatial refiner. DDT means Decoupled Diffusion Transformer, separating feature encoding from output prediction. Figure 1's caption and Section 3.3 now define and cite this design and distinguish its RAE/RAEv2 implementation. Real paired thumbnails show K23 codec inputs/reconstructions and K7 native RGB/adapter outputs, with checkerboards indicating transparency. Section 3.3 and Appendix D give the architecture and training details.
3/3: Data, evaluation definitions, measured cost, and ethics
Local TRAIN/VAL/TEST and the external set. The local corpus contains 8,635 RGBA assets. “Real-TRAIN” means its 7,779 training assets, used for codec training and real supervision of the adapter. The 422 validation assets serve development/model selection; the 434-image reconstruction test is excluded from fine-tuning. The 4,499 generated pseudo examples form a separate adapter-training pool.
The separately acquired external release is AlphaVAE's AlphaTest. It has 902 images: 500 from AIM-500 (Automatic Image Matting-500), introduced by Li, Zhang, and Tao, IJCAI 2021, and 402 Internet images. Exact-file screening against all local splits removes one AIM and 401 Internet matches, leaving AIM499 plus one Internet image. The latter passes the same exclusion rule and remains in the external evaluation, outside the AIM-only means. These exclusions concern the external evaluation pool, not half of the local training corpus. We also disclose that later aggregate RGB-conditioned recovery scores on the local test informed adapter development. Section 4.1 now defines and cites these sources, and Table 1 summarizes their roles.
Definitions and evaluation. We replaced the ambiguous expressions you identified:
- Background is a constant sRGB color broadcast across the image. Different foreground/alpha pairs can produce the same white composite, so that composite cannot uniquely determine opacity.
- “Continuous images” now means floating-point pixel values before byte quantization or JPEG. The matched codec protocol applies the same preprocessing, nine backgrounds, JPEG procedure, and metrics to all compared codecs.
- Alpha lies in . Alpha L1 is dimensionless; both means and difference intervals marked are scaled by 1000.
- “Both configurations” means AlphaRAE K7-256 and K23-1024, each compared with its resolution-matched AlphaVAE reference. The soft-interior result compares K23 with AlphaVAE at 1024: on 58 GT-defined external soft-interior images, unscaled regional Alpha L1 falls from AlphaVAE's 0.04026 to K23's 0.02627.
- The original reconstruction Table 1 is now Table 2 because of the added data-role table. The result tables are explicitly referenced in the evaluation/reconstruction text; direct-alpha definitions and units accompany Table 3.
How were the loss weights selected? The coefficients 1000/200/50/80/6 were chosen empirically in preliminary runs, informed by existing reconstruction objectives. The final recipe remained fixed throughout the reported formal 50k-update runs. Section 3.2 now states this selection process and the floating-point training convention.
Why do parameter count and measured cost differ? Parameter count measures stored weights, while latency depends on the executed operations and feature resolutions. Peak allocation also includes memory used during execution. At 1024 on one A800 in FP32 with batch one, we measure:
| Measurement | AlphaRAE | AlphaVAE |
|---|---|---|
| Codec parameters, including frozen encoder | 723.5M | 83.8M |
| Complete-path latency (ms) | 625.3 | 658.3 |
| Codec-only latency (ms) | 588.7 | 622.7 |
| Allocation after loading (GiB) | 2.7048 | 0.3218 |
| Additional runtime allocation (GiB) | 0.7856 | 3.0176 |
| Peak allocation (GiB) | 3.4904 | 3.3394 |
AlphaVAE's larger additional allocation offsets its smaller weight footprint at the peak. Complete-path timing includes preprocessing, device transfers, encode/decode, and output conversion, excluding file I/O, loading, and metrics. Section 4.1 explains these distinctions, and Appendix A gives the protocol and breakdown.
Human-study review and consent. We did not seek institutional IRB review or an exemption determination for this questionnaire. The revised Ethics Statement explicitly reports this status and the procedure. The 53 student volunteers completed an anonymous, voluntary online comparison of generated-image pairs. Before starting, they were informed of the study's purpose and research use of their responses. No names, student IDs, email addresses, or other personal information were collected. Small random-amount red envelopes were distributed anonymously through a messaging app as appreciation, without linking recipients to responses. Only aggregate preferences are reported.
1/3: Purpose, contributions, and concrete manuscript revisions
Thank you for the detailed reading and concrete examples. The original presentation compressed several explanations into short statements and left important terms undefined. We have rewritten the narrative across the abstract, method, experiments, discussion, and conclusion. We address the purpose and contribution here, the representation and training in Part 2, and the data, evaluation, and ethics in Part 3.
What problem does AlphaRAE solve? We want to reuse pretrained RGB representations and generators to create RGBA assets whose foreground color and opacity remain useful on new backgrounds, for image editing, graphic design, and composition. A white-background RGB composite cannot uniquely determine the underlying foreground and opacity. The codec preserves this additional information, and the adapter supplies the codec's inputs from an existing RGB generator.
Why evaluate reconstruction as well as generation? Reconstruction tests whether a representation preserves an existing asset's foreground color and opacity when decoded and composited onto different backgrounds. The codec also defines the latent targets and decoder used for adapter training. At generation, no target RGBA asset exists: the RGB generator supplies its native semantic latent and RGB pixels, but the RGBA decoder also needs the added detail component. Good reconstruction from encoded RGBA inputs therefore does not establish compatibility with generated RGB inputs. We evaluate these two capabilities separately.
The three contributions are now stated explicitly.
The reusable design principle is to preserve pretrained RGB semantic features, add detail for the target property, and learn a residual mapping from generated RGB inputs. We demonstrate this principle for opacity, as now stated in the conclusion.
How has the writing changed? We revised the surrounding explanations as well as the phrases you identified:
Figure 1 is also referenced when the method is introduced and in Section 3. The main text now explains the components and key configurations directly, with the appendices supplying extended protocols and analyses.