- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AlphaAnything presents a method for extending pretrained RGB generators to RGBA generation. The method retains and freezes the original RGB VAE, adapts the generative Transformer using LoRA, and adds a decoder-side alpha branch to produce RGB and opacity from the same generated latent, without separately training an RGBA VAE. The paper also introduces Endpoint-Aware Alpha Loss (EA Loss), which combines a Fisher–Rao opacity coordinate, Fenchel–Young score supervision, and separate square-root aggregation of foreground- and background-weighted contributions. Experiments cover text-to-RGBA and image-conditioned RGBA generation across multiple backbones. The results are competitive overall, with particularly notable improvements on the image-conditioned task.
Critical strengths
-
The motivation to avoid training an RGBA VAE is reasonable and practically relevant. Adapting an RGB VAE to RGBA requires additional representation learning and training design, and may alter its original RGB reconstruction capabilities. By freezing the RGB VAE and adding an alpha branch that receives RGB features unidirectionally, the paper offers a feasible approach to reusing pretrained representations.
-
EA Loss is motivated by the specific properties of opacity. The alpha channel includes exact transparency and opacity as well as continuous partial transparency. Compared with directly applying MSE, EA Loss more explicitly accounts for small residual errors near the endpoints, incorrect saturation, and localized detail errors. Its theoretical design is therefore well motivated.
-
The method achieves promising results and demonstrates applicability across backbones. Experiments with multiple pretrained models and both conditional generation tasks support the feasibility of the framework. The quantitative results and qualitative examples on the image-conditioned task also demonstrate good opacity prediction and compositing quality.
Critical weaknesses
-
A direct comparison with an alternative that also avoids training an RGBA VAE is missing. A central motivation of the paper is to preserve the original RGB VAE, but its implementation still introduces a trainable alpha branch. Another approach worth evaluating is to reuse the same frozen VAE to encode RGB and alpha separately, after replicating alpha into three channels, concatenate the two latent representations along the DiT's sequence dimension, and adapt the model for joint generation using LoRA (to my knowledge, this training approach works well). This approach is related to Zippo's dual-latent formulation. However, Zippo uses the Stable Diffusion 2.1 UNet and concatenates the latents along the channel dimension. Its architecture differs from sequence concatenation in a modern DiT, so its results do not directly establish the effectiveness or training difficulty of the latter approach.
-
The computational overhead of EA Loss during diffusion training and its effect on the final sampled outputs require further evaluation. Each evaluation of EA Loss requires converting the DiT's velocity prediction into an estimated clean latent , then processing this estimate through the RGB decoder's feature hierarchy and the alpha branch to compute the supervision. Although the original RGB decoder parameters are frozen, gradients must still propagate through the decoder to update the DiT LoRA. This supervision therefore adds decoding and gradient-propagation costs. The small reported inference overhead does not establish the additional training cost; comparisons of training time and GPU memory against training with only the latent-domain generative loss are needed. Furthermore, EA Loss is applied to a one-step estimate of at a randomly sampled noise level, rather than to the final output of the full multistep sampling process. The pixelwise properties of EA Loss still hold for this estimate, but those properties alone do not guarantee corresponding improvements in the final sampled outputs.
Questions for authors
- Please add an ablation experiment as discussed in Critical Weakness 1.
- Please explain why EA Loss remains appropriate for diffusion training.
6 comments
Rebuttal to Reviewer BUm2 (Q2)
EA Loss.
We appreciate the reviewer’s question about our opacity supervision based on a clean latent estimated in one step. The estimate can indeed differ from the final latent obtained through multistep sampling, but applying opacity supervision to the multistep final output at every training update would require running a multistep sampling and propagating gradients through multiple denoising steps. This can substantially increase computation and requiring additional activation storage or recomputation.
Our design already accounts for the varying reliability of this estimate by loss reweighting, while keeping opacity supervision computationally practical.
Given the model’s velocity prediction , the clean latent estimate is , and we apply EA Loss to the alpha decoded from this estimate. We explicitly reduce the weight of this opacity supervision at high noise levels, where could be less reliable. The difference between and depending on noise level follows directly from the formulation: , thus .
So for a given error in the predicted velocity , the error in the estimated clean latent decreases as the noise level decreases. We therefore use loss reweighting (Sec. 3.4, Eq. (12)) to retain stronger opacity supervision at moderate and low noise levels, where is expected to be more accurate and closer to the final latent obtained through multistep sampling, and reduce the weight at high noise levels.
Similar strategies also appear in SpeeDiff (CVPR 2026) and ImageReward (NeurIPS 2023). Specifically, SpeeDiff applies losses to images decoded from , while the ReFL method in ImageReward applies reward feedback to clean sample predictions at selected late denoising steps.
The effect on final sampled outputs is evaluated directly by the stop-gradient ablation in Table 4: allowing EA Loss gradients to update the DiT through improves opacity accuracy and composite fidelity after multistep inference on both backbones.
Training overhead.
Table D quantifies the additional cost of extending backbone training to RGBA generation. “Backbone LoRA (reference)” uses the standard latent flow matching objective without alpha prediction or opacity supervision. It establishes the computational cost of backbone training under matched settings, allowing us to measure the additional cost of learning opacity.
Table D. Training cost under matched settings. Time per optimizer update is reported as mean ± standard deviation.
| Backbone / Task | Training configuration | Time / update (s) | Peak memory (GiB) |
|---|---|---|---|
| FLUX.1 T2RGBA | Backbone LoRA (reference) | 2.925 ± 0.071 | 36.84 |
| Zippo baseline | 6.435 ± 0.041 | 39.19 | |
| AlphaAnything | 3.840 ± 0.063 | 44.65 | |
| FLUX.2 T2RGBA | Backbone LoRA (reference) | 2.629 ± 0.042 | 36.86 |
| Zippo baseline | 5.768 ± 0.055 | 39.34 | |
| AlphaAnything | 3.588 ± 0.059 | 43.78 | |
| FLUX.2 I2RGBA | Backbone LoRA (reference) | 5.750 ± 0.002 | 39.35 |
| Zippo baseline | 9.872 ± 0.001 | 41.83 | |
| AlphaAnything | 6.694 ± 0.002 | 44.86 |
Relative to this reference, AlphaAnything adds 0.915 s-0.959 s per optimizer update across the three settings, with 5.51–7.81 GiB of additional peak memory. This manageable additional cost enables training for RGBA generation capability on the pretrained backbones.
For the comparison between RGBA generation methods, AlphaAnything requires much less time per optimizer update than the Zippo baseline in all three settings. AlphaAnything generates color and opacity from one shared latent, retaining the original generated sequence length, whereas Zippo adds a separate sequence of alpha latent tokens that the DiT processes at every update. As reported in our response to Q1, our training and samping time savings are achieved while maintaining competitive T2RGBA performance and achieving better results on five of six metrics on both RGBA-Image and RefMatte-RW100.
We hope our responses have addressed your concerns. If you have any remaining questions or concerns, we would be glad to provide further clarification and address them.
Dear Reviewer BUm2,
We sincerely thank you for your time and effort in reviewing our work and for your valuable feedback.
As the discussion period draws to a close, we would like to briefly summarize how our rebuttal addresses your main concerns:
- Following your suggestion, we added a Zippo baseline that reuses the same frozen RGB VAE to encode RGB and alpha separately and concatenates their latents along the DiT sequence dimension; AlphaAnything achieves competitive T2RGBA performance, better results on five of six metrics on both RGBA-Image and RefMatte-RW100, and lower inference latency than this baseline.
- We clarified the rationale for EA supervision on one-step clean-latent estimates and its noise-dependent weighting, and highlighted the stop-gradient ablation showing that propagating this supervision to the DiT improves opacity accuracy and composite fidelity after full multistep sampling.
- We quantified the training overhead against the backbone LoRA reference: 0.915–0.959 s per optimizer update and 5.51–7.81 GiB of additional peak memory, and showed that AlphaAnything requires less training time per update than Zippo.
We would be happy to provide further clarification and answer any remaining questions. If you feel our responses have addressed your concerns, we would appreciate it if you could consider raising your score.
Rebuttal to Reviewer BUm2 (Q1)
We sincerely thank the reviewer for your time and effort in reviewing our work, as well as for your constructive and valuable feedback.
Zippo presents a compelling approach to RGBA generation by reusing a pretrained RGB VAE to encode alpha maps as three-channel grayscale images. It jointly generates an RGB image and a grayscale alpha map through separate latent representations, enabling the modeling of color and opacity.
AlphaAnything explores a different perspective: pretrained RGB generators can learn competitive RGBA generation capabilities without RGBA reconstruction pretraining or a separate alpha latent. In AlphaAnything, RGB and opacity are generated from a single shared latent, with an alpha branch reading features from the frozen RGB decoder. During training, EA Loss trains the alpha branch and propagates gradients through the frozen decoder into the DiT LoRA, allowing opacity supervision to shape the shared generative latent.
Following your suggestion, we implemented a Zippo baseline with a separate alpha latent on the same DiT backbones, concatenating RGB and alpha latents along the token dimension. For the Zippo baseline, we use the same training settings as AlphaAnything, including the LoRA configuration. We evaluate both methods on RGBA-Text, RGBA-Image, and RefMatte-RW100 (Referring Image Matting, CVPR 2023), using the same evaluation settings and metrics as in our paper.
Table A. T2RGBA evaluation on RGBA-Text.
Table B. I2RGBA evaluation on RGBA-Image and RefMatte-RW100.
AlphaAnything achieves better results on five of six metrics on both RGBA-Image and RefMatte-RW100, while obtaining comparable results on RGBA-Text. These findings support our central insight that competitive RGBA generation does not require a separate latent representation for alpha. We also provide qualitative comparisons with the Zippo baseline for T2RGBA and I2RGBA at anonymous link (T2RGBA) and anonymous link (I2RGBA), respectively.
Table C. Inference cost for T2RGBA and I2RGBA. We report inference time per image and peak memory using the same inference setting on a single GPU.
Additionally, during inference, AlphaAnything's can generate RGBA images while remaining nearly identical time cost of the original RGB generator, and is 2.16×, 2.09× and 1.60× faster than Zippo on FLUX.1 and FLUX.2 T2RGBA and FLUX.2 I2RGBA. These results demonstrate that generating RGB and alpha from a shared latent achieves competitive quality while substantially reducing inference latency, making it particularly attractive for large scale applications.