acceptodds
Under review as a conference paper at ICLR 2027

VQ-DSGAN: Mitigating Latent Representation Collapse via Dual-Space Discrimination

Abstract

Vector-quantized tokenizers are a key interface for modern visual generation, yet their effectiveness is often limited by latent representation collapse. In practice, this failure appears not only as codebook collapse, where a small subset of codewords dominates usage, but also as dimensional collapse, where quantized latents occupy a narrow low-rank subspace despite a high nominal code dimension. Existing remedies are largely quantizer-centric and do not explicitly couple balanced code usage with image-space distinguishability and latent-space recoverability. We propose VQ-DSGAN, a shared energy-based adversarial framework for standard learned-codebook VQ tokenizers that addresses these failure modes in a unified manner. VQ-DSGAN leverages a single VQ-VAE backbone in generator and discriminator modes via mode-specific feature-wise affine modulation, and induces a bidirectional dual-space discriminator from the same discriminator-mode subnetwork: an encoder-decoder energy for image-space discrimination and a decoder-encoder energy for latent-space discrimination. The image-space objective provides dense, structurally grounded supervision for perceptual fidelity, while the latent-space objective combines a code-centered prior with noise-injected latent recovery to encourage balanced code usage and locally invertible latent geometry. Experiments show that VQ-DSGAN improves reconstruction fidelity and downstream class-conditional generation while achieving near-complete codebook utilization and efficiently addressing dimensional collapse without changing the standard learned-codebook VQ formulation.

Primary area:
generative models
Keywords:
Vector-Quantized Generative Models, Codebook Collapse, Dimensional Collapse, Discrete Latent Representation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_D7qQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes VQ-DSGAN to address codebook underutilization and dimensional collapse in learned vector-quantized tokenizers. It shares an encoder–decoder backbone between generator and discriminator modes through FiLM-style modulation, combining image reconstruction energies with latent recovery energies under a code-centered prior. Experiments on ImageNet 256×256 report improved reconstruction and class-conditional generation, supported by ablations of the discriminator design, target prior, and latent noise.

Critical strengths

1.The shared bidirectional discriminator offers a coherent way to combine image fidelity and latent recoverability without redesigning the quantizer. The FiLM implementation adds few parameters and introduces no additional discriminator computation at inference.

2.The component ablations are informative. In Table 6, removing noise preserves 98% utilization but reduces effective dimension from 128 to 32 and worsens rFID from 1.26 to 1.54. This supports the practical value of addressing code usage and latent geometry separately.

Critical weaknesses

1.The theory does not establish the dimensional-collapse property measured experimentally. Section 4.1 measures the PCA spectrum of codebook vectors, whereas Theorem 9 concerns the local Jacobian of the discriminator-mode decoder–encoder map. Distinct codewords can all lie on a line while this map is the identity, satisfying the local recovery condition despite rank-one codebook covariance. Moreover, invertibility of individual FiLM transformations does not establish equivalent geometry across the complete generator and discriminator networks. These gaps weaken the paper’s central mechanistic explanation, although they do not invalidate the reported improvements.

2.The generation comparison does not isolate the tokenizer’s contribution. Appendix D.2 also strengthens the autoregressive generator through adaptive conditioning and other refinements, but Table 2 provides no comparison holding the generator, training budget, and sampling settings fixed. This matters given the relatively small FID gains over IBQ. Reproduction is further hindered by unspecified training durations, the exact 604M architecture, and selected noise/EMA hyperparameters.

Questions for authors

1.What additional argument connects Theorem 9’s local Jacobian result to codebook PCA effective dimension, and establishes transfer between generator and discriminator modes? 2.Do the generation gains persist when VQ-DSGAN and IBQ tokenizers use the same autoregressive architecture, training budget, and sampling protocol? 3.Please specify the training durations, exact 604M configuration, sampling settings, EMA decay, and chosen noise coefficient.

Reviewer_pNQ6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VQ-DSGAN. The model transforms the encoder and decoder of a VQ-VAE into an energy-based discriminator and employs a mechanism to modulate FiLM parameters within GroupNorm layers. To address the issues of codebook collapse and dimensional collapse, it introduces adversarial energy terms in both the image space and the latent space. Using this approach, the model achieves an 1.26 rFID and 99% codebook utilization on the ImageNet 256x256 dataset. Also for class-conditional generation, it records FID scores of 2.72 and 2.34, depending on different model sizes.

Critical strengths

The authors did thorough ablations, and some results are promising. Utilization increases from 6% to 99% when using both branches, and effective dimension increases when plugged into LlamaGen, IBQ and VQGAN-LC (Table 7). Also the author stated training overhead openly.

Critical weaknesses

More evidence is needed to show that adversarial shared-backbone is effective. In the latent branch real and fake are both noisy codewords, and in the Code-Prior Matching ablation they have identical distribution. So there's no adversarial game there but effective dimension still increases from 8 to 85 as shown in Table 3 and 6.

This look like a autoencoding penalty robuts to noise, and a uniform usage term, so non-adversarial control experiment is needed. Also theory also doesn't really cover what's measured or the implemented objective.

It is hard to distinguish the advantages of each component with the current generation comparison experiments. The second stage use strengthened AR model which uses AdaLN, top-k, CFG, and baselines use their own recipes. The gains over IBQ are small - 2.72 vs 2.88, 2.34 vs 2.45, tested with single seed. Also, K=16,384 and K=100,000 give identical rFID LPIPS Eff. Dim (1.26/0.12/128) in Table 1, which looks odd.

Questions for authors
  1. Can you add a simple non-adversarial control (usage-entropy penalty plus a noise-augmented code reconstruction loss) on the same backbone, and does it get most of the gains in Table 3/6?
  2. What happens if you train the exact same AR model on tokens from IBQ, VQGAN-LC, LlamaGen and your tokenizer, with a few seeds?
  3. Why are rFID, LPIPS and effective dimension identical for K=16,384 and K=100,000 in Table 1, and is that a saturation or a copy error?.
Reviewer_h8GdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VQ-DSGAN to address both codebook collapse and dimensional collapse in VQ-based image tokenizers. Instead of modifying the quantizer, the method shares the VQ-VAE backbone between generator and discriminator, using feature-wise affine modulation to separate the two modes. It introduces a dual-space discriminator, where image-space discrimination improves reconstruction quality, and latent-space discrimination uses a code-centered prior and noise-injected recovery to improve codebook usage and latent representation.

The experiments on ImageNet show improvements in reconstruction quality, codebook utilization, effective dimension, and downstream image generation compared with existing methods. The ablation studies also show that the image-space and latent-space components have different but complementary effects.

Critical strengths

The paper addresses both codebook collapse and dimensional collapse, which I think is an important problem for VQ-based tokenizers. I like that the authors do not only focus on codebook utilization, since high utilization does not necessarily mean the latent representation is good. The ablation studies in Tables 3 and 6 provide useful evidence for the contribution of different components.

The experimental results are strong. With a codebook size of 16,384, the method achieves 99% utilization and an effective dimension of 128, which is much higher than the compared methods. The reconstruction and downstream generation results are also competitive. I also appreciate the experiments on different VQ frameworks in Appendix F, which show the method can potentially be applied to other tokenizers.

Critical weaknesses

My main concern is about the novelty of the method. Several components, including energy-based adversarial learning, latent prior matching, and feature-wise modulation, have already been studied before. The main contribution seems to be combining these existing techniques in a shared framework. Although the results are good, I am not fully convinced that the proposed design is necessary. A comparison with simpler non-adversarial regularization methods would make the contribution more convincing.

I also have some concerns about the theoretical analysis. In Appendix B.5, the authors analyze dimensional collapse through the local Jacobian rank of the decoder-encoder mapping. However, the experiments measure effective dimension using PCA on the learned codebook. I think these two are not directly equivalent, and the current analysis does not fully explain why the proposed method should improve the global effective dimension. This makes the theoretical support for the main claim less convincing to me.

Questions for authors

Can the authors compare the proposed method with a simpler approach that combines codebook usage regularization and noise-injected latent reconstruction, without adversarial training? I think this would help clarify whether the improvement mainly comes from these regularization terms or the dual-space adversarial design.

Can the authors explain more about the connection between the local Jacobian rank in Theorem 9 and the PCA-based effective dimension used in the experiments? Does the local invertibility guarantee a higher global effective dimension, or is this mainly supported by the empirical results?

For the downstream generation results in Table 2, can the authors provide a comparison using the same autoregressive generator architecture and training setting with different tokenizers? Since the compared models have different generator configurations, it is difficult to tell how much of the improvement comes specifically from the tokenizer.

Reviewer_xmmzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses codebook collapse (few codes used) and dimensional collapse (codebook vectors span a low-rank subspace) in learned-codebook VQ tokenizers. It proposes VQ-DSGAN, which replaces the patch-based GAN discriminator with an energy-based one built from the tokenizer itself: one VQ-VAE backbone serves as both generator and discriminator through per-channel affine (FiLM-like) parameters at each GroupNorm layer. Two energies are derived from the discriminator mode, an image-space reconstruction-plus-LPIPS energy and a latent-space decoder-encoder energy. The latent branch contrasts noisy code centers drawn from a uniform prior with noisy quantized latents, with the aim of balancing code usage and keeping the latent space locally invertible. On ImageNet 256x256, the paper reports 99% codebook utilization, effective dimension 128 (versus 4-32 for baselines), rFID 1.26, and class-conditional AR generation FID 2.72 (313M) and 2.34 (604M). Ablations cover the two branches, the discriminator design, and the latent prior, and a plug-in study applies the module to VQGAN-LC, IBQ and LlamaGen.

Critical strengths

A parameter-light design with a convincing mechanistic ablation. Deriving both critics from the tokenizer's own backbone (about 0.04M extra parameters) is a neat idea that leaves the quantizer and inference unchanged. Tables 3 and 6 show the role of each part: removing the latent branch drops utilization from 99% to 33% and effective dimension from 128 to 8, and removing noise injection alone collapses effective dimension to 32. Table 4 controls for discriminator capacity.

Large, consistent gains in the collapse metrics, with signs of generality. At 256-d codes the method reaches effective dimension 120-128 against 28-32 for IBQ, with the best reported rFID/LPIPS (1.26/0.12). Attaching the module to IBQ, VQGAN-LC and LlamaGen also raises their effective dimension (Table 7), suggesting the effect is not tied to one architecture.

Critical weaknesses

The downstream claims are not isolated to the tokenizer, and the benefit of the collapse metrics is not shown. Appendix D.2 reveals that the VQ-DSGAN generator is a different AR backbone (VAR-style AdaLN, top-k, CFG, qk-norm) from the baselines in Table 2, and the margins over IBQ are small (FID 2.72 vs 2.88; 2.34 vs 2.45) and could come from the generator or sampling settings. Baseline numbers appear to be copied from the papers, and the K=16,384 and K=100,000 rows of Table 1 are identical. More fundamentally, utilization and effective dimension are diagnostics, and the evidence does not show that raising them improves results. A 100,000-entry codebook with full utilization gives no reconstruction gain over 16,384 (rFID 1.26, LPIPS 0.12 in both). In Table 3, removing only the latent discriminator leaves effective dimension at 8 yet gives a better rFID (1.87) than removing only the image discriminator (2.15, effective dimension 100). Table 7 reports only utilization and effective dimension, with no rFID or FID. A tokenizer with an 8-d code and effective dimension 8 (LlamaGen, rFID 2.19) also reconstructs well. So the gain in rFID/FID may come from the image-space branch or training details rather than from higher effective dimension.

Missing baselines and a gap between the theory and what is measured. SimVQ, VQBridge/FVQ, DCVQ and distribution matching are positioned against in the introduction but absent from the tables, as is any cheap non-adversarial alternative (a uniform-usage regularizer plus decoder-encoder consistency on noisy codes). This matters because in Table 6 the "w/o code-prior matching" variant is effectively a self-consistency regularizer, yet already raises effective dimension from 8 to 85. Since a uniform target directly optimizes utilization and perplexity, reporting them as evidence is partly circular. The theory (Appendix B) concerns idealized objects (a non-parametric discriminator, a freely movable code distribution, a local Jacobian at codewords) rather than the PCA spectrum of the codebook that is measured. Results come from single runs, and all evidence is on ImageNet 256x256 with class-conditional generation only. The paper itself notes the uniform prior may not suit data with non-uniform semantic frequency, which is the typical large-scale setting.

Questions for authors

In the latent branch, what exactly is fed to the discriminator-mode decoder-encoder for real and fake samples: a single vector, or a 16x16 grid in which each position holds an independent noisy code center (real) or the noisy quantized latent of an image (fake)? If grids of random codes are used, how does the network behave on spatially incoherent inputs?

Please report the tokenizer training schedule, whether the baselines in Tables 1-2 were re-trained or copied, the LPIPS protocol, and the CFG, top-k and sampling settings of all generators. Why are the K=16,384 and K=100,000 rows identical?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.