acceptodds
Withdrawn from ICLR 2027

Short Circuit VAE: Using Pre-Trained Variational Autoencoders For Latent Upscaling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scaling modern diffusion models to high resolutions introduces a computational bottleneck that grows quadratically () with the spatial dimensions and frequently degrades structural realism. Pixel-space super-resolution (SR) models inherently require heavy architectures to process uncompressed RGB data, and risk semantic drift when generating fine details from scratch. Existing latent upscalers either require slow, multi-step denoising loops, or rely on external architectures that ignore the pre-trained features of the base model, demanding extensive training and often producing smoothed outputs. To overcome these limitations, we introduce Short-Circuit VAE (SC-VAE). Instead of building a complex upscaler from scratch, SC-VAE acts as a shortcut directly inside the pre-trained Variational Autoencoder (VAE). Specifically, we reuse the early layers of the VAE decoder to extract features and spatially upscale the latents. These upscaled features are passed through a small, trainable adapter, and then fed into the final layers of the VAE encoder to project them back into the standard latent format. By relying on the VAE's own boundary layers to format the data, our method safely upscales images while remaining fully compatible with the generative model.

Primary area:
generative models
Keywords:
latent image super-resolution, latent upscaling, Variational Autoencoders, high resolution image generation, image super-resolution, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gpsWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article introduces the Short-Circuit VAE (SC-VAE), which upsamples 2×/4× in the latent space by reusing some layers of a pre-trained VAE decoder and encoder, thus avoiding extra iterations or patch-wise denoising in high-resolution diffusion. The method uses a two-stage training in both latent space and pixel space, and experiments are conducted on FLUX, FLUX2, and SD3.5. The results show that SC-VAE can achieve competitive high-resolution generation quality at a lower inference cost, and ablation studies also support the effectiveness of its architecture design and training strategy.

Critical strengths
  1. The structure is simple and efficient. It uses the internal layers of a pre-trained VAE for latent upscaling, avoiding extra high-resolution diffusion steps and giving a clear boost in computational efficiency.
  2. The experiments are pretty thorough. They cover multiple diffusion/VAE architectures and include ablation studies and user studies, which somewhat back up the method's effectiveness.
Critical weaknesses
  1. The novelty compared to existing methods isn't very clear. LUA already uses single-pass latent upscaling and similar multi-stage training strategies, so this paper's main new idea seems to be just reusing internal VAE layers. The paper lacks a strict comparison of parameter counts, training costs, and model capacity, so it doesn’t fully prove that this design brings real technical improvements.
  2. The fairness of the 4K experiments needs clarification. The 4K test set in the Supplementary comes from MultiAspect-4K-1M, which UltraFlux also used for training. So it needs to be clear if the test images were fully separate from UltraFlux’s training data, otherwise the comparisons in Table 1 might be contaminated. Also, SC-VAE doesn’t outperform UltraFlux on some 4K quantitative metrics.
Questions for authors

Are the 1,000 images in the 4K test set completely excluded from UltraFlux's training data? Please explain the specific data split.
How many trainable parameters and total training compute does SC-VAE have? Is it possible to make a fair comparison with LUA in terms of parameter count/compute?
Why wasn't there a comparison with recent methods specifically aimed at high-resolution latent decoding/upscaling (like PiD)?
The paper says the VAE is frozen, but the training section mentions that the decoder upscaling block is fine-tuned. Please clarify which parameters are frozen and which ones are trained.

Reviewer_yTqoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Short-Circuit VAE (SC-VAE). It reuses the early part of a pre-trained VAE decoder and the late part of the encoder, and inserts lightweight residual adapters in between for latent upsampling. This turns high-resolution generation from "directly sampling at high resolution" into "low-resolution generation + single-step latent upscaling."

Critical strengths
  1. The paper does not attempt to overturn the mainstream latent generation paradigm, but instead targets the most painful part of the current pipeline — high-resolution upscaling — making the method naturally well-suited to the real-world needs of systems like FLUX and latent diffusion models.
  2. The authors provide not only quantitative results at 2K/4K, but also qualitative comparisons, user studies, and ablation experiments; the FLUX2 version achieves top-tier results at both 2K and 4K in the paper's main table, and scores the highest average ratings in realism and text alignment in the user study.
  3. The paper keeps most pre-trained VAE layers frozen, only training the adapters and fine-tuning a small number of VAE-related modules, which makes it more like a plug-and-play high-resolution enhancement module rather than a method that requires retraining the entire main generation model.
  4. Compared with multi-step latent refinement or pixel-space super-resolution baselines, SC-VAE retains both a speed advantage and strong visual quality through a single forward pass, demonstrating a favorable quality-speed trade-off.
Critical weaknesses
  1. Novelty is more engineering than conceptual. This work feels more like a "high-quality system re-organization" than a "new problem formulation." SC-VAE still heavily relies on existing VAE/latent pipelines, so the conceptual novelty is somewhat limited.
  2. Tightly coupled to specific base models. SC-VAE is closely tied to the specific backbone it is built on. The paper trains separate versions based on FLUX1 and FLUX2 VAEs, and attributes some performance differences to the expressive power of different latent spaces. This suggests the module is more of a VAE-specific adapter than a truly model-agnostic, general-purpose high-resolution component.
  3. Pipeline optimization, not a paradigm shift. In terms of "whether it truly solves the high-resolution scaling problem," the paper is more about pipeline optimization than a fundamental change. While SC-VAE is strong in results and practicality, it does not provide equally strong evidence on memory scaling.
  4. Limited to upscaling, not a full generation framework. Looking at its capability boundary, SC-VAE acts more like a "high-quality upscaler" than a complete new generation framework. The paper itself admits that it cannot fix structural errors already made by the base model — does this mean its gains on global composition and high-level semantics are limited?
Questions for authors

see above comments

Reviewer_gNooOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SC-VAE connects early VAE decoder layers to the encoder’s final projection through a trainable adapter for latent upscaling. It uses staged latent- and pixel-space training and evaluates 2K/4K generation across several base models.

Critical strengths

The design is simple and practical. It avoids additional denoising and reports lower end-to-end latency than LUA, with better patch-level metrics. This offers a useful quality–speed tradeoff.

Critical weaknesses
  • I’m not yet convinced that reusing VAE layers is what makes the method better. LUA already does single-pass latent upscaling with staged latent/pixel training. The specific connection may be new, but we still need a comparison using the same data, losses, and training budget to see what it adds. Patch metrics improve, while the 4K FID on FLUX1 is almost the same as LUA: 59.41 versus 59.49. The current ablations do not fully separate the architecture’s benefit from the training recipe (§4.5).

  • Some of the quality results are hard to interpret. The best results use FLUX2 without a matched FLUX2 baseline, so it is unclear how much the upscaler itself helps. There is also a mismatch in the user study: Table 2 reports 4.12 for text alignment, while the four category averages in Table 6 range from 4.06 to 4.10. A weighted average should not exceed all category averages. This may be a reporting error, but it needs clarification, along with confidence intervals.

Questions for authors
  • How much does VAE reuse help when the inputs, data, losses, and training budget are kept the same? Could you compare against a similarly sized standalone network and report its upscaling time?

  • In Standard Init, did you freeze the randomly initialized VAE layers or train them? Did each ablation get the same training budget?

  • Which base model did you use for the user study? How did you calculate the overall scores, and what explains the mismatch between Tables 2 and 6? Could you provide corrected results with confidence intervals?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.