acceptodds
Withdrawn from ICLR 2027

RAC: Rectified Flow Auto Coder

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In this paper, we propose a **R**ectified Flow **A**uto **C**oder (**RAC**) inspired by Rectified Flow to replace the traditional VAE: **1.** It achieves multi-step decoding by applying the decoder to flow timesteps. Its decoding path is straight and correctable, enabling step-by-step refinement. **2.** The model inherently supports bidirectional inference, where the decoder serves as the encoder through time reversal (hence *Coder* rather than encoder or decoder), reducing parameter count by nearly 41%. **3.** This generative decoding method improves generation quality since the model can correct latent variables along the path, partially addressing the reconstruction–generation gap. Experiments show that RAC achieves a Pareto improvement over SOTA VAEs, where even a 10× parameter-reduced decoder exceeds full-scale VAE performance in both reconstruction and generation quality, validating the effectiveness of our approach.

Primary area:
generative models
Keywords:
Rectified Flow, Flow Matching, Variational Autoencoder

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_FjaUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a multi-step VAE decoder that uses rectified flow to decode latent to image. The decoder works as an encoder by reversing the flow direction and encoding images down to latents. The central motivation to do this is to support generation quality by allowing the VAE to potentially fix a one-step latent to image process that might have errors from imperfect generated latents. As a consequence of this pipeline, the encoding-decoding process requires less parameters because the decoder can act as an encoder.

Critical strengths

S1: The paper proposes the novel concept of decoding latents through a continuous path instead of a one-step latent to image jump.

Critical weaknesses

W1: The paper needs to be redesigned with significant effort on writing and presentation of the text. L267 has "generation quality" folded into the reconstruction quality section in several areas where it is not possible to pin down what is being discussed. For reconstruction quality, the authors seem to use global metrics like rFiD instead of per-sample metrics that are important to discuss. Per sample metrics are sidelined into the appendix, and even then do not compare the default RAC against its own SD-VAE teacher. There is no LPIPS anywhere either. As a paper focused on improving VAE quality, this seems to be a poor set of experiments to evaluate quality.

W2: The paper discusses 5 different components for loss computation but never ablates any of them. How did the authors come upon 4 different hyper-parameters for training? The ablation discusses some configurations but they are nowhere enough.

W3: The paper does not attempt to scale image generation or reconstruction beyond 256 px resolution. Higher resolutions might be slower or more computationally intensive at multi-step decoding. 256px resolution is already slower than the SD-VAE. Finally, quality needs to be consistent across resolutions and some form of scaling is required in the experiments.

W4: The paper claims that latents are being improved but trains in the same latent space with the same encoder teacher. How is the generation v/s reconstruction gap handled without changing the latent space?

Questions for authors
  1. 4 sign-language papers from Fang et al. are cited but appear to be off-topic. Please justify or remove.
  2. See weaknesses.
Reviewer_JotVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

RAC replaces the VAE decoder with a multi-step velocity field from a block-repeated latent to the image, and uses reverse integration as the encoder. It claims better gFID than REPA-E, 41% fewer parameters, and a 0.1× decoder that beats SD-VAE.

Critical strengths
  1. Turning the VAE decoder into a lightweight, time-conditioned multi-step decoder is a reasonable direction, and it fits the current interest in generative decoders. Because the teacher encoder is frozen, RAC stays compatible with existing KL-VAE latent spaces and can be attached to pretrained generators without retraining them.
  2. The controlled decoder swap on a frozen DiT-XL (Table 7) shows consistent gains (gFID 7.13 → 6.74–6.98) while using fewer decoder GFLOPs (94.3–254.5 vs 310.4). The paper also reports latency (3.66× SD-VAE) and a latent-perturbation test (Table 6), which make the compute–quality trade-off easier to judge.
Critical weaknesses
  1. App. C.4 acknowledges that "Published baseline entries follow the configurations reported by their originating studies" (p. 18, l. 951–953), but in REPA-E the first rows of Tables 1 and 2 are SiT trained with vanilla REPA for 100K iterations (20 epochs), with the note "All baselines are reported using vanilla-REPA". RAC instead labels them "SD-VAE" and "baseline SiT models Ma et al. (2024)" (p. 7, l. 364–365), and neither table states the training budget (only C.4 gives "100K-iteration (20-epoch)" for SiT-XL, l. 943–944). App. C.4 also states that "+RAC" "replaces the decoder branch" with a decoder trained against "the frozen baseline encoder" (l. 947–949), yet SiT-L moves from 24.1 to 14.7 (Table 1, l. 241–242). This is far more than the decoder-only swap on a frozen DiT-XL in Table 7 ("7.13 to 6.98", p. 17, l. 905–913), and it also beats REPA-E's end-to-end tuning (16.3). If the SiT is truly unchanged, a decoder whose rFID improves from 0.62 to 0.35–0.44 (Table 1 right) would have to account for a 9.4-point gFID gain, which should be supported by further explanation. C.4 says the "training budget" is matched (l. 951) but not whether the +RAC rows were trained with REPA alignment or whether the SiT was retrained. Without this, it is hard to tell how much of the improvement comes from the decoder itself.
  2. The abstract claims "the model can correct latent variables along the path" (p. 1, l. 029) and Fig. 3 says the method "theoretically aims for perfect reconstruction" (p. 3, l. 151), but a deterministic multi-step decoder sees only the latent and has no information about what the correct latent would be. The "40.82% reduction" (App. B.1, p. 16, l. 834) counts an encoder that generation does not use, since "the attached backbone ... supplies zT" (p. 15, l. 809). The robustness gains in Table 6 (p. 17, l. 864) could equally come from a smoother decoder. Likewise, the state reconstruction in Fig. 11 ("only 100 training steps", p. 8, l. 395) maps image to state and back with no bottleneck, so its high quality is expected rather than informative. Reporting gFID with a backbone trained on RAC-encoded latents would show whether the RAC encoder is useful in practice.
Questions for authors
  1. Tables 1–3 (p. 5, l. 236; p. 7, l. 363; p. 8, l. 424) and App. C.4 (p. 18, l. 935–949): were the baseline rows re-run or taken from REPA-E, and which backbone, iteration count and REPA setting do the +RAC rows use?
  2. Table 1 (p. 5, l. 236): what are the results of decoder-only swaps in the REPA-E 100K setting?
Reviewer_MQaoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces RAC, which addresses the generation-reconstruction performance gap through a shared continous-time velocity field, and hence could achieve the performance of traditional multi-step VAE while supporting bidirectional inference at the same time. The simulation shows much smaller parameter counts with similar performance.

Critical strengths
  1. It is insightful to propose shared velocity field for decoder and encoder to resolve the reconstruction-generation gap.
Critical weaknesses
  1. While the idea might be intersting, the overall presentation, from abstract, figure captions, to preposition abuse and layout arrangement, need improved organization, which affects readibility of this paper. Also, the paper jumps between words like latents, states, and generator-produced representations, making their reference hard to follow.

  2. The reviewer feels a gap between claimed contribution and verification, i.e., the abstract and methods highlight correcting generator-induced latent mismatch, yet the simulation seems to show improved performance with higher efficiency (i.e., parameter count). How generation-reconstruction alignment is proved? Fig. 11 compares decoding teacher-encoder latents with full-state reconstruction, and the former does not directly involve generator-produced latents, and the latter retains a different information budget.

  3. Many figures are hard to read. For example, Fig. 10 is titled “Encoding and Decoding Trajectory Symmetry,” but the reviewer could not distinguish forward and reverse trajectories. Similarly, Figure 6 is labeled as state reconstruction but discussed in relation to detail preservation under spatial compression.

  4. The novelty also needs more justification. Iterative generative decoding is already established, and the distinctive contribution is potentially the shared bidirectional formulation and its compatibility with pretrained KL-VAE latents. However, the current experiments do not sufficiently isolate the benefits of these features from additional decoding computation and regularization.

Questions for authors
  1. While the idea might be intersting, the overall presentation, from abstract, figure captions, to preposition abuse and layout arrangement, need improved organization, which affects readibility of this paper. Also, the paper jumps between words like latents, states, and generator-produced representations, making their reference hard to follow.

  2. The reviewer feels a gap between claimed contribution and verification, i.e., the abstract and methods highlight correcting generator-induced latent mismatch, yet the simulation seems to show improved performance with higher efficiency (i.e., parameter count). How generation-reconstruction alignment is proved? Fig. 11 compares decoding teacher-encoder latents with full-state reconstruction, and the former does not directly involve generator-produced latents, and the latter retains a different information budget.

  3. Many figures are hard to read. For example, Fig. 10 is titled “Encoding and Decoding Trajectory Symmetry,” but the reviewer could not distinguish forward and reverse trajectories. Similarly, Figure 6 is labeled as state reconstruction but discussed in relation to detail preservation under spatial compression.

  4. The novelty also needs more justification. Iterative generative decoding is already established, and the distinctive contribution is potentially the shared bidirectional formulation and its compatibility with pretrained KL-VAE latents. However, the current experiments do not sufficiently isolate the benefits of these features from additional decoding computation and regularization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.