acceptodds
Under review as a conference paper at ICLR 2027

Learning Discrete Representations via Mutually Exclusive Random Variables

Abstract

We propose a novel approach, named Mutually Exclusive Random Variable Autoencoder (MERVar-AE), to learn discrete representations based on Mutually Exclusive Random Variables (MERVar). Mutually Exclusive Random Variable (MERVar) arises from the observation that deterministic mappings (e.g., neural decoders) involving random variables (noise injection) tend to reduce overlap coefficients among involved random variables, thereby inducing exclusivity. This exclusivity can be utilized to learn binary latents. Specifically, by injecting symmetric noise into the output of a bounded activation function (e.g., ), the output values are pushed toward the boundaries ( or for ), effectively producing binary latent representations. Compared to previous approaches based on Straight-Through Estimator (STE), such as VQ-VAE, the proposed approach is simpler, with only an activation and noise injection operation. The forward and backward computations are aligned, as gradients are propagated through the same continuous latent representation. Comprehensive experiments on nine standard datasets show that the proposed approach achieves competitive results compared to state-of-the-art methods in discrete representation learning. Controlled experiments further demonstrate the benefit of avoiding surrogate-gradient mismatch during encoder optimization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
discrete representation learning, binary latent representations, autoencoders, noise injection, STE-free quantization, mutually exclusive random variables

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xmGROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies image codes with entries ±1\pm 1 entries. An encoder output is passed through tanh, so each coordinate lies in (−1,1)(−1, 1), then the Gaussian noise is added, and the encoder and decoder are trained to reconstruct the image. The claim is that the noise pushes different images apart, so coordinates move toward ±1\pm 1. Once enough coordinates are near ±1\pm 1, the noise is turned off. At this stage each coordinate is replaced by its sign, the encoder is frozen, while decoder not. Freezing the encoder avoids a gradient through the sign. There is theoretical argument on overlapping noise clouds. The paper also includes experimental validation of the method and comparison with existing approaches.

Critical strengths

The paper uses a well-known and effective idea, i.e., to inject noise so that the codes become nearly binary. Under a shared training setup, the experiments show that the method performs well.

Critical weaknesses

MERVar-AE method hardly can be claimed to be novel, as it follows the mechanism proposed in [1]. It has already been proposed to add to add Gaussian noise, for example, at a saturating code layer and minimize reconstruction error, so that intermediate activations become unreliable and the codes saturate (and then threshold). [2] use that method on images and report that rounding is as effective. The related-work paragraph of this manuscript cites [1] and presents MERVar as the mathematical account of the same phenomenon. The account does not add any new mechanism.

I’d like to note that the proposed overlap coefficient is basically one minus total variation, i.e., OC$(p, q) = 1 - TVTV(p, q)$. In this regard, the claims of paper become even clearer. In particular, Theorem 3.5 is the standard formula for the total variation of two equal isotropic Gaussians followed by the chain rule.

Table 3 separates the definition from the reported run. With tanh, the maximum overlap at 16K updates is about 0.91, so the codes are not ϵ\epsilon-mutually exclusive for any ϵ<0.9\epsilon < 0.9. Removing tanh is what drives the maximum overlap to order 10−410^{-4}. Thus, saturation and mutual exclusivity, as defined, seem to occur in different ablations.

Table 7 only ranks methods under a short shared budget (50K CIFAR-10 updates, then 10K updates on the target set). On ImageNet the second stage uses about 300K images. This is the whole comparison with VQ-VAE, FSQ, LFQ, BSQ, and the other listed methods. I think it is not evidence about those methods as large-scale tokenizers, which is the reading invited by "representative state-of-the-art quantization techniques".

Minor points. The factor 1/21/2 in (9) comes from quadratic inequality for ∥u−v∥2\|u - v\|^2 which is 2-strongly convex in the target vv. Remark A.1 does not say in which argument the loss is strongly convex, and strong convexity in the prediction does not yield (9). Independence of the noise is unused in both proofs and cannot affect a marginal overlap. Unimodality and symmetry are unused in Theorem 3.3, they belong only to the non-Gaussian claim in Remark 3.6, which is not proved. That remark cites Appendix A.3.3 as empirical support, however Table 11 in Appendix A.3.3 reports reconstruction PSNR, not distances or overlap coefficients.

[1] Salakhutdinov, Ruslan, and Geoffrey Hinton. "Semantic hashing." International Journal of Approximate Reasoning 50.7 (2009): 969-978.

[2] Krizhevsky, Alex, and Geoffrey E. Hinton. "Using very deep autoencoders for content-based image retrieval." ESANN. Vol. 1. 2011.

Questions for authors
  1. In Table 3, the mean overlap falls to about 0.009, but the largest overlap stays near 0.91, and the definition 3.1 is a bound on the largest overlap. Which number do you use to show that training follows Theorems 3.3 and 3.5?

  2. Table 7 uses 50K CIFAR-10 updates and then 10K updates on the target set. Could you add larger-scale experiments, closer to how VQ-VAE, FSQ, LFQ, and BSQ are usually trained, to support the comparison with representative state-of-the-art quantization methods?

3 comments
AuthorsOpenReview ↗

MERVar Theory, Overlap Statistics, and Large-Scale Validation

Thank you for recognizing the method's effectiveness under a shared training setup. We address the novelty, theoretical scope, and experimental-scale concerns below.

 

W1: Contributions Beyond the Shared Mechanism

The connection to semantic hashing is already discussed in Related Work. Our principal contribution is the formulation and mathematical analysis of MERVar, particularly the MSE Lower Bound Theorem (Theorem 3.3), which relates reconstruction error to distance-weighted overlap. The Mutual Exclusivity Theorem (Theorem 3.5) further characterizes Gaussian pairwise separation. Beyond this analysis, we develop and evaluate STE-free binary image tokenization for autoregressive generation, whereas semantic hashing targets retrieval and its 2011 image-retrieval extension uses an STE-type estimator. A shared mechanism alone does not negate these theoretical, optimization, and empirical contributions, supported by our controlled experiments, downstream generation results (Table 14), and new large-scale comparisons.

 

W2: OC–TV Equivalence and the Role of Theorem 3.5

The identity OC⁡(p,q)=1−TV⁡(p,q)\operatorname{OC}(p,q)=1-\operatorname{TV}(p,q) is an equivalent representation, rather than an alternative to our analysis. We use OC because it directly quantifies the overlap responsible for reconstruction ambiguity and yields a natural expression of the MSE lower bound. It also makes the definition of ε\varepsilon-MERVar transparent: OC⁡(pi,pj)≤ε\operatorname{OC}(p_i,p_j)\leq\varepsilon directly bounds pairwise overlap, equivalently TV⁡(pi,pj)≥1−ε\operatorname{TV}(p_i,p_j)\geq 1-\varepsilon. Rewriting the bound using 1−TV⁡1-\operatorname{TV} leaves its content unchanged and does not establish the bound itself.

The Mutual Exclusivity Theorem (Theorem 3.5) uses the Gaussian overlap formula and the chain rule to characterize the separation direction induced by overlap minimization. Its role is to make this optimization property explicit within the MERVar framework. This observation does not subsume the MSE Lower Bound Theorem (Theorem 3.3), which establishes the relationship between reconstruction error, distributional overlap, and pairwise target distances. The known OC–TV identity and Gaussian formula therefore do not negate this additional theoretical contribution.

 

W3 / Q1: Separation Dynamics and Overlap Statistics

Table 3 demonstrates the progressive separation of latent distributions during training, rather than claiming that every pair has reached a small overlap. With tanh, the mean OC decreases from approximately 0.990.99 to 0.0090.009, while the maximum OC of approximately 0.910.91 reports the remaining worst-case overlap. These statistics describe the separation dynamics; they are not used to claim uniform satisfaction of Definition 3.1 for a small ε\varepsilon.

Figure 4 directly evaluates the distance-weighted overlap lower bound in Theorem 3.3 against reconstruction MSE; neither the unweighted mean nor the maximum alone represents that bound. Theorem 3.5 characterizes the separation direction under pairwise Gaussian overlap minimization, without guaranteeing uniformly small overlap for all pairs during reconstruction training. The no-tanh ablation in Table 3 further examines how removing the bounded latent constraint affects separation.

 

W4 / Q2: New Large-Scale Tokenizer Comparisons

Table 7 evaluates methods under identical encoder–decoder architectures and a shared training budget; it isolates behavior under controlled conditions rather than reproducing each method's largest configuration. Our new large-scale results address the separate question of competitiveness with large tokenizers.

 

Minor Points: Proof Assumptions and Non-Gaussian Scope

Strong convexity. The relevant argument in Remark A.1 is the target: for fixed prediction yy, v↦∥y−v∥2v \mapsto \lVert y-v\rVert^2 is 22-strongly convex. Equivalently, Eq. (9) follows directly from

∥y−xi∥2+∥y−xj∥2=2∥y−xi+xj2∥2+12∥xi−xj∥2.\lVert y-x_i\rVert^2 + \lVert y-x_j\rVert^2 = 2\left\lVert y-\frac{x_i+x_j}{2}\right\rVert^2 + \frac{1}{2}\lVert x_i-x_j\rVert^2.

We will specify the argument explicitly; no convexity of the neural network is required.

Noise assumptions. Theorem 3.3 uses sufficient assumptions consistent with our model, although its proof only requires marginal densities. The Gaussian assumption in Theorem 3.5 enables an explicit overlap formula and separation direction. We will clarify these roles.

Non-Gaussian behavior. Table 11 evaluates the applicability of MERVar-AE across different noise distributions through reconstruction performance. It is not intended as a proof of non-Gaussian separation. Theorem 3.5 concerns isotropic Gaussian noise; the additional experiments demonstrate the method’s empirical applicability beyond this setting. We will clarify this distinction in Remark 3.6.

I have seen the responses given by the authors to my and other reviews. I appreciate their clarification. I have raised my score and I assume that the refinement of the manuscript will be conducted (including theoretical part).

AuthorsOpenReview ↗

Thank you for the discussion

Thank you for considering our responses and raising your rating. We will include the semantic hashing and large-scale tokenizer comparisons, clarify the theorems’ assumptions and their connection to reconstruction, and explain how the mean overlap in Table 3 differs from the maximum in Definition 3.1.

Reviewer_RsR8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an autoencoder approach for predicting binary representations, called Mutually Exclusive Random Variable (MERVar). This field has a fundamental problem in that gradients are disconnected during the binarization step, so proxy methods such as the straight-through estimator have been proposed, as in VQ-VAE.

This work proposes a formulation that preserves gradients without such gradient proxies, and the key components are the noise injection in Eq. 1 and the tanh activation function. The formulation appears to be based on the obvious observation that the overlap between representations decreases as a proper encoding model is trained. By using the tanh function, the authors expect the representations to concentrate at the two extremes, -1 and +1. It uses hard truncation only during the inference.

Surprisingly, this simple method improves reconstruction metrics over existing methods.

Critical strengths
  1. The proposed method achieves improvements in reconstruction error in Table 9 through a very simple and intuitive approach, and the gap appears to be highly significant. In particular, it is good to see improvements in LPIPS, a perceptual metric, as well.

  2. Figure 6 shows that the tanh values indeed concentrate at the two extremes.

  3. Figure 7 suggests that the latent space is even semantically meaningful.

Critical weaknesses
  1. The paper lacks comparisons evaluating whether the model truly learns semantically meaningful representations, such as linear classification performance, rather than only reconstruction quality. Looking at Table 1 of DiTi (ICLR 24) and Table 1 of DBAE (ICLR 25), recent related papers seem to place more emphasis on linear classification performance.

  2. The presentation quality is very low. In particular, there are too many tables in the main text, and it would be better to present some of them as figures. For example, Table 3 could be presented as a plot showing performance across updates, and Table 6 could be presented as a trend plot with respect to the number of codebooks.

  3. What are the advantages of discrete representations over continuous representations? I am not sure whether being able to train a latent AR model is a clear advantage. Doesn't continuous diffusion already work well in the image domain? I understand the importance of discrete representations, but the paper would be more convincing if it included an application where these advantages are demonstrated. Alternatively, a comparison with continuous representation models would be helpful.

  4. The theoretical justifications provided in Theorem 3.3 and Theorem 3.5 are not sufficient to explain why the latent representations converge to binary values. Theorem 3.3 only provides a lower bound on the reconstruction error based on the overlap between distributions, while Theorem 3.5 shows that minimizing pairwise overlap increases the distance between latent centers. However, neither theorem guarantees that optimizing the actual reconstruction objective causes the latent values to move toward −1 or +1 rather than converging to intermediate values. This is an important limitation of the theoretical justification for the proposed method.

Yue, Z., Wang, J., Sun, Q., Ji, L., Chang, E., & Zhang, H. (2024, May). Exploring diffusion time-steps for unsupervised representation learning. In International Conference on Learning Representations (Vol. 2024, pp. 8209-8219).

Kim, Y., Lee, K., Park, M., Na, B., & Moon, I. C. (2025, May). Diffusion bridge autoencoders for unsupervised representation learning. In International Conference on Learning Representations (Vol. 2025, pp. 84846-84887).

Questions for authors

See weakness.

7 comments
AuthorsOpenReview ↗

Representation Utility, Presentation, and Theoretical Scope

Thank you for recognizing the simplicity of our approach, the reconstruction improvements, and the meaningful latent structure illustrated in Figures 6 and 7. We address your four concerns below.

 

W1: Semantic Evidence and Downstream Evaluation

As the reviewer notes, Figure 7 suggests meaningful structure in the latent space. Figure 9 further visualizes recognizable structures in the binary latents, while Table 14 evaluates downstream autoregressive image generation. Our evaluation therefore extends beyond reconstruction quality.

DiTi and DBAE study diffusion-based continuous representation learning, with linear attribute prediction evaluating the accessibility of semantic information. Our focus is STE-free binary image tokenization for reconstruction and generative modeling. Linear probing would provide complementary quantitative evidence, but evaluates a different property from the generative utility tested in Table 14. We distinguish the qualitative evidence in Figures 7 and 9 from a quantitative claim of linear semantic separability.

 

W2: Main-Text Organization and Complementary Visualizations

We respectfully disagree with the overall characterization of the presentation as “very low”: the main text prioritizes the core theory and quantitative evidence, while complementary visualizations are provided in the appendix, including training curves (Figures 4 and 8), latent separation (Figure 5), binarization dynamics (Figure 6), latent-space interpolation (Figure 7), and meaningful structure in the binary representations (Figure 9). Tables 3 and 6 retain exact values for numerical inspection. However, we appreciate the suggestion that plots could make their trends easier to follow.

 

W3: Utility of Discrete Representations

Our objective is to develop an effective way to learn binary image tokens without STE-based encoder optimization. The success of continuous diffusion does not eliminate the utility of discrete tokenization, which provides finite-alphabet representations with an explicit bit budget and supports categorical autoregressive modeling. Table 14 demonstrates this application using the learned binary tokens. We do not claim that discrete representations universally outperform continuous ones.

Matching latent dimensions does not match representation capacity between continuous and binary latents. A meaningful rate-constrained comparison requires specifying the precision or quantization of continuous latents. Our controlled evaluation therefore focuses on discrete tokenizers with matched bit budgets and architectures. The new large-scale comparisons in our general reply further demonstrate competitive reconstruction against modern image tokenizers, including WeTok and BSQ.

 

W4: What the Theory Establishes About Binarization

This distinction is already explicitly discussed in Remark 3.6 (Lines 225–230). The MSE Lower Bound Theorem (Theorem 3.3) establishes a necessary reconstruction constraint through distributional overlap weighted by pairwise target distances. The Mutual Exclusivity Theorem (Theorem 3.5) characterizes how minimizing pairwise Gaussian overlap increases latent-center separation. As explained in Remark 3.6, since the latent centers are bounded within [−1,1]D[-1,1]^D, this separation provides a theoretical motivation for boundary-seeking behavior. Importantly, Remark 3.6 explicitly states that this is a tendency rather than a guarantee of coordinate-wise convergence to binary values. We do not claim a global convergence guarantee under the actual reconstruction objective.

This boundary-seeking tendency is further supported empirically: Figure 6 tracks the concentration of latent values near ±1\pm1 together with reconstruction quality, while Figure 8 shows the evolution of the binary rate and loss. Section 6 discusses cases where binarization may be insufficient or optimization may stall. Finally, hard binarization is not merely an inference-time operation. As described in Section 4 and Table 2, we apply it after continuous training, freeze the encoder, and further optimize the decoder on the binary codes.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.

Addressed concerns:

  • Semantic representations (W1): The additional linear probing experiments on CIFAR-10 and CelebA-HQ provide meaningful evidence of the learned representations.

Remaining concerns:

  • Presentation (W2): The presentation could still be improved by replacing some tables with more intuitive visualizations.

  • Discrete representation utility (W3): While the downstream AR generation results are encouraging, the advantages over continuous representations remain unclear. In particular, I would be interested in how the linear probing results compare with continuous representation learning methods such as DiTi and DBAE under comparable evaluation settings.

  • Theoretical justification (W4): The authors clarify that their theorems do not guarantee convergence to binary values. However, the connection between optimizing the reconstruction objective and the observed convergence toward ±1 remains theoretically unexplained.

Nevertheless, I still find the empirically observed tendency of representations to converge toward −1 and +1 interesting. I consider this a novel approach to learning binary representations, despite the remaining limitations. Therefore, I will maintain my original rating (Weak Accept).

AuthorsOpenReview ↗

W3: Linear Probing Results and Comparison with DiTi/DBAE

Thank you for your positive assessment and for recognizing the novelty of our approach.

Regarding W3, we would like to draw your attention to the CelebA-HQ 40-attribute classification results already presented in our General Reply, which directly address your request for comparisons with DiTi and DBAE. For convenience, we summarize these results below.

CelebA-HQ 40-attribute classification (AP)

We train a 40-way BCE linear probe on frozen MERVar-AE binary latents, using CelebA-HQ 256×256 (28K train / 2K test) and the official CelebA labels. The encoder was trained exclusively on CIFAR-10 and has never seen face images.

RepresentationProbe input dimMean AP (%)
Chance level (mean positive rate)–24.9
MERVar-AE binary latent (10×64×64)40,96064.3
MERVar-AE binary latent, 8×8 average-pooled64064.0

Per-attribute AP (%).

HighestMale 99.3No_Beard 98.0Wearing_Lipstick 97.2Eyeglasses 95.9
Smiling 94.8Young 94.0Mouth_Slightly_Open 92.2Heavy_Makeup 91.8
LowestBlurry 8.8 (4 positives)Wearing_Necklace 25.9Narrow_Eyes 28.5Straight_Hair 31.1

Reference: published transfer results. We report the FFHQ-trained setting from DiTi/DBAE Table 1, where encoders are trained on a separate dataset before CelebA attribute probing.

MethodRepresentationEncoder training dataProbe dataMean AP (%)
IB-GAN512-d continuousFFHQCelebA42.8
β-TCVAE512-d continuousFFHQCelebA43.2
PDAE512-d continuousFFHQCelebA59.7
Diff-AE512-d continuousFFHQCelebA60.5
SimCLR512-d continuousFFHQCelebA60.8
DiTi512-d continuousFFHQCelebA61.4
DBAE-d512-d continuousFFHQCelebA65.6
DBAE512-d continuousFFHQCelebA66.4
MERVar-AE (ours)10 binary channelsCIFAR-10CelebA-HQ64.3 / 64.0 (640-d)

These published results serve as context rather than a direct comparison, since the datasets and evaluation protocols differ. Notably, FFHQ → CelebA transfer remains within the face domain, whereas our CIFAR-10 → CelebA-HQ transfer crosses substantially different domains.

Nevertheless, our reconstruction-only binary representations achieve comparable AP, even after spatial pooling to 640 dimensions, providing further evidence of their semantic utility.

We appreciate your remaining concerns regarding presentation and theoretical justification and will address them in the revision. We hope this additional evidence further strengthens your positive assessment of our work, and we would sincerely appreciate your consideration of a higher rating.

Thank you again for your constructive feedback and consideration.

Reviewer_Jf72OpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2(+1)
Read review
Paper summary

MERVar-AE learns binary latent representations by applying tanh to the encoder output and adding Gaussian noise to it during training.

Critical strengths
  1. Simple design, using tanh's natural constraints to achieve a binarized latent representation
  2. Comprehensive analysis of ablations of the binarization threshold, activation function, and noise type.
Critical weaknesses
  1. The mechanism is known. 1.1 The method is similar to semantic hashing; the additional contribution is limited 1.2 Using noise to replace STE is standard in image compression and variational methods
  2. The method's benefit looks like a small-scale effect. The gain comes from a more exploited latent capacity, and the encoder and decoder are strong enough to make use of this additional capacity. The gain is likely from the fact that small baseline autoencoders are underoptimized or misaligned in terms of the encoder, decoder, and latent capacity. It's likely the same gain can be achieved by simply adding nodes in the bottleneck.
  3. The binary latents' value is unknown. Other than reconstruction, there is no linear probing or other downstream task that can showcase the benefit of the binary coding. And even for reconstruction, the gain from binary code cannot be decoupled from simply increasing bottleneck capacity.
Questions for authors
  1. How do the performance/inference speeds of MERVar-AE compare to an AE/VAE with increased bottleneck capacity?
4 comments
AuthorsOpenReview ↗

Novelty, Capacity Utilization, and Downstream Validation

W1.1: Theoretical Novelty of MERVar and Practical Novelty of Image Tokenization

As detailed in our general reply, our contributions include the formulation and mathematical analysis of MERVar, particularly the MSE Lower Bound Theorem (Theorem 3.3) and the Mutual Exclusivity Theorem (Theorem 3.5), and its application to STE-free binary image tokenization for autoregressive generation (Table 14). The connection to semantic hashing is already discussed in Related Work; a shared mechanism does not negate these additional theoretical and empirical contributions. Its 2011 image-retrieval extension also uses an STE-type estimator. Our controlled experiments examine the effects of STE-based encoder updates in MERVar-AE (Table 5), BSQ (Table 12), and VQ-VAE without commitment regularization (Table 13).

 

W1.2: No Prior Work Identified Supporting the “Standard” Characterization

We conducted an extensive literature search on neural image compression and variational methods but did not identify prior work using the same noise-induced separation mechanism to learn binary image tokens without STE-based encoder optimization. The three most relevant works we found illustrate the distinction:

[1] applies tanh followed by stochastic hard binarization. Its “noise” expresses a random choice between −1 and +1, with gradients passed unchanged through the binarizer using an STE-type estimator. It does not use continuous additive noise to drive latent representations toward binary endpoints.

[2] replaces scalar quantization with additive uniform noise during training, followed by multi-level quantization and entropy coding at inference. The noise approximates quantization; it does not induce binary separation in a bounded latent space.

[3] first learns continuous representations using a noise-based quantization approximation, then fine-tunes the decoder with hard quantization and a frozen encoder. This also differs from using noise to induce binary separation.

[1] Variable Rate Image Compression with Recurrent Neural Networks. ICLR, 2016.

[2] End-to-end Optimized Image Compression]. ICLR, 2017.

[3] Soft then Hard: Rethinking the Quantization in Neural Image Compression. ICML, 2021.

Our method targets image tokenization for generative modeling, and our comparisons focus on image tokenizers rather than image compression, which additionally use entropy coding to reduce the transmitted bitrate. Our principal contribution is the formulation and mathematical analysis of MERVar, neither of which is established in these works. Nor do these works evaluate this mechanism as an STE-free binary image tokenizer against modern state-of-the-art generative image tokenizers. The established use of noise for quantization approximation therefore does not negate our theoretical contribution or empirical advances.

 

W2: Fixed-Budget Capacity Utilization and Benefits Beyond Small Models

Our controlled comparisons use identical autoencoder architectures, bottleneck dimensions, bit budgets, and training budgets. The gains therefore do not arise from adding bottleneck units. More effective use of latent capacity under the same constraints is itself a substantive benefit of a tokenizer. Increasing the baseline’s bottleneck would instead allow it to retain more bits, changing the comparison constraint. Our new large-scale results also address the small-model concern. As shown in our general reply, the MERVar-AE+ surpasses WeTok on all four ImageNet reconstruction metrics at the same 0.5 bpp, with model training on a single RTX 4080 (16 GB).

 

W3: Value of Binary Tokens and Downstream Generation

Table 14 evaluates downstream autoregressive image generation, directly testing the learned binary tokens in a generative task. Figure 9 also provides qualitative evidence that the binary latents preserve recognizable structures. Our controlled comparisons use identical architectures, latent dimensions, bit budgets, and training budgets (lines 799- 815). Improving reconstruction under these matched constraints is precisely the benefit of the proposed approach. Increasing the baseline’s bottleneck would change the bit budget and therefore does not explain the gains observed in our fixed-budget comparisons.

 

Q1: Inference Speed and Matched-Budget Comparisons

Comparing discrete representations with different bottleneck capacities does not provide a matched-budget evaluation. Inference speed depends on the specific encoder–decoder architecture and computational cost, rather than bottleneck capacity alone. Our controlled comparisons use identical architectures and bit budgets, and their standard encoding and decoding speeds are approximately the same.

AuthorsOpenReview ↗

Additional Response to W1.2, W2, and W3

 

We would like to further address your concerns after carefully reviewing the comments several times.

 

W1.2: Learning discrete latent representations without using the straight-through estimator (STE) remains a challenging task, since discrete operations such as rounding are non-differentiable. STE is therefore a widely adopted and straightforward approach to addressing this problem. Starting from VQ-VAE [1], which has received thousands of citations and continues to attract increasing research attention, numerous extensions have been proposed, with STE remaining a widely adopted approach for learning discrete latent representations.

As discussed in Table 1 and Related Work, to the best of our knowledge, only a limited number of methods learn discrete representations without STE, including Gumbel-Softmax, SQ-VAE [2], TokenBridge, and our noise-based approach. In particular, none of these methods, except our proposed approach, has sufficiently demonstrated competitive reconstruction performance against modern STE-based methods such as WeTok [3].

It is worth noting that avoiding STE is not a trivial improvement. Our controlled experiments (Tables 5, 12, and 13) demonstrate that STE-based encoder optimization can substantially degrade reconstruction performance. Specifically, introducing STE into MERVar-AE reduces the test PSNR by approximately 1.5 dB (Table 5), while a similar degradation of approximately 1 dB is observed in BSQ (Table 12). Furthermore, VQ-VAE exhibits severe degradation when the commitment loss is removed (Table 13). These results suggest that the forward-backward gradient mismatch introduced by STE can negatively affect encoder optimization, and highlight the importance of avoiding STE.

 

W2: In our comparisons, all methods are evaluated under identical settings, including network architecture, latent capacity, training updates, optimizer, and learning rate. The only difference is the quantization method. Furthermore, all methods are trained for 50k updates, which is significantly longer than the default training schedules used in the original implementations of the compared methods. We also observed that all methods reached stable convergence. Additional evidence is provided by the code usage and entropy results reported in Tables 15–23.

Therefore, we believe that the improvements are unlikely to result from underoptimized or misaligned baseline autoencoders. Under identical experimental settings, the observed performance differences should primarily come from the quantization methods themselves. Moreover, the ability to more fully utilize a fixed latent capacity is itself an advantage of a quantization method. Higher code usage and entropy also provide supporting evidence for more effective utilization of the available latent space, which can contribute to better reconstruction accuracy. Under identical training conditions and a fixed latent capacity, more effective capacity utilization should be considered an advantage rather than a limitation.

 

W3: We are not sure whether the reviewer is questioning the value of binary latent representations in general or specifically the binary latents learned by our proposed method. Binary latent representations are one of the simplest and most efficient forms of discrete representations. Discrete representations have well-established applications in text, image, and audio modeling, compression, and generative modeling. In particular, discrete tokens serve as the foundation of modern autoregressive models, including GPT-based language models. Similar to BSQ, LFQ, and the recent WeTok [3], our proposed approach learns the same type of binary latent representations, and we have provided direct comparisons with these methods. Furthermore, we provided a downstream generation task and achieved better results.

Regarding the criticism that "the gain from binary code cannot be decoupled from simply increasing bottleneck capacity," we are confused by this concern and would appreciate further clarification.

All comparative experiments reported in our paper are conducted under a fixed latent capacity. The reconstruction improvements cannot be attributed to an increase in bottleneck capacity, since the compared methods use the same latent dimensions and bit budgets.

While increasing the bottleneck capacity may improve reconstruction quality, achieving better reconstruction under the same bit budget is precisely one of the fundamental objectives of discrete representation learning and image compression. Simply increasing the number of latent nodes would change the bit budget and violate the principle of fixed-capacity comparisons.

[1] Neural Discrete Representation Learning. NeurIPS, 2017.

[2] SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization. ICML, 2022.

[3] WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction. ICLR, 2026.

Thanks for the clarification and additional experiments. I personally worked on binary representations several years ago, and my main concern is the binary representation itself. Admittedly, human data tends to have binary representations due to their low cognitive burden, which makes facilitating binary representation in ANNs meaningful for fitting human data. However, to facilitate this, you need a special constraint (tanh activation in your case), and the change of constraint under constant capacity is almost always a trade-off; you gain some binary encoding power while losing some expressiveness in other aspects (non-binary representations, nuanced details, for example). Thus, how to define a real large scale use case to justify such a trade-off is important. And this is what I found lacking in the initial scripts. But I think this has been answered to a great extent by the added downstream classification task and large-scale reconstruction benchmark. It would make the paper more informative if you could show the limitations and the "loss of expressiveness in some other aspects"; it would make the paper more rigorous.

In general, I believe the rebuttal contains a large amount of content that should go into the paper. If the semantic hashing comparison, large-scale benchmark, and the downstream classification are integrated, I would recommend accepting the paper. Thus, I decided to raise my score, conditioned on all rebuttal content being integrated into the final version.

AuthorsOpenReview ↗

Thank you for the discussion

Thank you for taking the time to review our additional experiments and for raising your rating. We will include the semantic hashing comparisons, large-scale reconstruction benchmarks, and downstream classification results in the revised manuscript. We will also discuss the potential trade-off you highlighted between binary coding and the preservation of fine-grained detail at a fixed code length. We sincerely appreciate your support.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Reply

We would like to clarify our contributions and present new large-scale results before addressing each reviewer individually.
 

1. Novelty and Contribution: MERVar Theory and the MERVar Autoencoder

Our main theoretical contribution is the formulation and mathematical analysis of Mutually Exclusive Random Variables (MERVar), including the MSE Lower Bound Theorem (Theorem 3.3), which relates reconstruction error to overlap between noisy latent distributions, and the Mutual Exclusivity Theorem (Theorem 3.5), which characterizes pairwise latent separation. Both of them are the starting point for MERVar-AE, which learns binary image tokens without STE-based encoder optimization. As with modern tokenizers such as BSQ [3] and WeTok [4], which build on established binary quantization principles, our contribution should be assessed by the theoretical insights, optimization properties, and empirical advances—not dismissed on the basis of a shared mechanism alone. To the best of our knowledge, we are the first to systematically analyze this separation mechanism and apply it to STE-free binary image tokenization for autoregressive generation. Semantic hashing [1] was developed for document retrieval rather than generative image tokenization, and its 2011 image-retrieval extension [2] uses an STE-type estimator. Table 1 already distinguishes our approach from representative modern tokenizers: LFQ and BSQ [3] learn binary codes using STE, whereas the STE-free alternatives considered there—VP-VAE, SQ-VAE, and TokenBridge—follow different, non-binary formulations.

   

2. New Large-Scale Comparisons with State-of-the-Art Tokenizers

We trained a larger MERVar-AE on a single RTX 4080 (16 GB). We highlight the 0.5 bpp setting below:

DatasetMethodBppPSNR ↑SSIM ↑LPIPS ↓rFID ↓Params (M)Training hardwareTotal memory (GB)Training time (h)
ImageNetWeTok [4]0.50028.310.8490.0710.21447.88128 × H20 (96 GB)12,288NR
ImageNetBSQ [3]0.56328.630.8550.1010.3817416 × RTX A5000 (24 GB)384NR
ImageNetMGVQ [5]0.81330.100.8900.0550.31122NRNRNR
ImageNetMERVar-AE0.50028.380.8550.1033.62649.491 × RTX 4080 (16 GB)16≈336
ImageNetMERVar-AE+0.50029.130.8610.0640.10649.491 × RTX 4080 (16 GB)16≈336
COCOWeTok [4]0.50028.470.8650.0682.35447.88128 × H20 (96 GB)12,288NR
COCOBSQ [3]0.56328.640.8670.1033.2817416 × RTX A5000 (24 GB)384NR
COCOMGVQ [5]0.81330.320.9030.0602.30122NRNRNR
COCOMERVar-AE0.50028.400.8660.10810.03649.491 × RTX 4080 (16 GB)16NR
COCOMERVar-AE+0.50028.860.8650.0680.81649.491 × RTX 4080 (16 GB)16NR

MERVar-AE+ denotes latent optimization with all model parameters frozen, followed by binarization; it retains the same code length and decoder as MERVar-AE.

On ImageNet, MERVar-AE+ outperforms WeTok at the same bitrate and BSQ at a higher bitrate across all four metrics. On COCO, it achieves higher PSNR and lower rFID than both, while matching WeTok's SSIM and LPIPS. These results demonstrate effectiveness beyond the original small-model setting. Achieving competitive performance under such a substantial training-hardware disparity is challenging: our model is trained on a single RTX 4080 (16 GB), whereas WeTok reports 128 H20 GPUs (96 GB each)—768× the aggregate GPU memory capacity. Despite this disparity, MERVar-AE+ surpasses WeTok on all four ImageNet reconstruction metrics at the same bitrate, demonstrating the practical significance of our approach under limited training resources. We attribute these gains in part to two features of our approach: its simplicity and its directly differentiable continuous objective. MERVar-AE only uses reconstruction objectives, without commitment or adversarial losses. Its differentiable formulation further supports latent optimization with a fixed decoder, with no additional model training or change to code length or decoding cost. In contrast, applying STE-based latent optimization to our larger MERVar-AE decreases reconstruction quality, supporting the practical benefit of our differentiable formulation in this setting.

[1] Semantic hashing. International Journal of Approximate Reasoning, 2009.

[2] Using very deep autoencoders for content-based image retrieval. ESANN, 2011.

[3] Image and Video Tokenization with Binary Spherical Quantization. ICLR, 2025.

[4] WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction. ICLR, 2026.

[5] MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantization. arXiv:2507.07997, 2025.

AuthorsOpenReview ↗

Clarification of the Difference from Semantic Hashing

 

We further clarify a substantive difference between the computational formulations of semantic hashing and MERVar-AE.

 

Semantic hashing injects Gaussian noise before the sigmoid activation [1, 2], whereas MERVar-AE injects noise after the bounded activation:

MethodComputationNoise
Semantic hashingc=sigmoid⁡(a+ϵ)c=\operatorname{sigmoid}(a+\epsilon)ϵ∼N(0,σSH2)\epsilon\sim\mathcal N(0,\sigma_{\mathrm{SH}}^2)
Gumbel-Softmax / Binary Concrete (binary case) [4]c=sigmoid⁡((a+ℓ)/τ)c=\operatorname{sigmoid}((a+\ell)/\tau)ℓ∼Logistic⁡(0,1), τ>0\ell\sim\operatorname{Logistic}(0,1),\ \tau>0
MERVar-AEz~=tanh⁡(a)+σϵ\tilde z=\tanh(a)+\sigma\epsilonϵ∼N(0,I)\epsilon\sim\mathcal N(0,I)

 

These operations are not interchangeable. In semantic hashing, noise is transformed by sigmoid and its effect on the output can be attenuated through saturation as the unbounded input aa becomes strongly positive or negative. In MERVar-AE, the decoder directly receives additive-noise distributions centered at the bounded latent codes z=tanh⁡(a)z=\tanh(a), where aa is the encoder output.

 

In computational structure, semantic hashing is closer to the binary form of Gumbel-Softmax / Binary Concrete [4]: both perturb logits before applying a sigmoid that acts as a smooth binary threshold, although their noise distributions and training procedures differ. The 2011 image-retrieval extension [3] instead rounds the sigmoid outputs during the forward pass and ignores rounding during backpropagation, using an STE-type estimator to train through the hard threshold. In contrast, MERVar-AE perturbs the post-activation latent representation and avoids STE-based encoder optimization, a key contribution whose practical benefits are supported by our controlled experiments.

 

The different noise placement also leads to different explanations of binarization. Semantic hashing offers a qualitative argument that pre-activation noise encourages sigmoid saturation. Our MERVar analysis instead examines the post-activation latent distributions, establishing a reconstruction-error lower bound in terms of their overlap and pairwise target distances, and characterizing Gaussian pairwise separation.

 

Overall, MERVar-AE differs from semantic hashing in both its theoretical formulation and its computational design; it is not merely a new interpretation of the same computational mechanism.

 

[1] Semantic hashing. International Journal of Approximate Reasoning, 2009. Section 3.2, p. 973 (PDF p. 5), first paragraph, second sentence: noise is added to “the bottom–up input received by each code unit.”

[2] Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. Bengio et al., 2013. Section 2, p. 3, paragraph immediately after Eq. (1): “in semantic hashing (Salakhutdinov and Hinton, 2009) noise is added (just before the non-linearity).”

[3] Using Very Deep Autoencoders for Content-Based Image Retrieval. ESANN, 2011. Section 2.1, PDF p. 3, first paragraph: “we rounded the outputs of the logistic units” during the forward pass, but “ignored this rounding during the backward pass.” This describes an STE-type estimator, although the paper does not use that name.

[4] The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. ICLR, 2017. Section 4, p. 7, explicitly describes semantic hashing as adding Gaussian noise to logits before the logistic function. Appendix B, p. 14, Eq. (16), gives the Binary Concrete sampling formula used above.

AuthorsOpenReview ↗

Response on Linear Classification (Reviewer RsR8 W1, Reviewer Jf72 W3)

RsR8 W1: The paper lacks comparisons evaluating whether the model learns semantically meaningful representations, such as linear classification performance, rather than only reconstruction quality (cf. Table 1 of DiTi, ICLR'24, and DBAE, ICLR'25).

Jf72 W3: Other than reconstruction, there is no linear probing or other downstream task that can showcase the benefit of the binary coding.

We thank the reviewers for this suggestion and added frozen-feature linear-probe experiments following DiTi/DBAE. The probe is a single linear layer trained on the hard discrete latents of a frozen encoder: ±1 bits for MERVar/BSQ/LFQ, quantized levels for FSQ, selected codewords for VQ-style methods, and one-hot indices for GS. No method uses continuous features. No hyper-parameter is selected on the test set.

1. CIFAR-10 linear classification (50K train / 10K test)

MethodProbe input dimProbe paramsTop-1 (%)Top-3 (%)
Raw pixels (reference)307230.7K38.3969.47
TokenBridge640.65K31.8061.09
GS65536655K34.0565.74
FSQ3203.2K39.6971.48
LFQ6406.4K40.0372.32
RQ-VAE5125.1K40.5071.28
BSQ6406.4K41.1972.51
SQ-VAE6406.4K42.0873.60
VQ-VAE409641.0K42.3874.66
VP-VAE102410.3K43.6975.53
LooC409641.0K45.3677.10
MERVar-AE (ours)6406.4K44.1475.16

Observations.

  • Same 640-d input. MERVar-AE beats SQ-VAE by +2.06, BSQ by +2.95 and LFQ by +4.11 in top-1. BSQ and LFQ are the STE-based binary quantizers.
  • Larger probes. It also beats VQ-VAE (+1.76) and is within 1.2 points of LooC, even though both use a 6.4× larger probe.

The absolute accuracies are modest for all methods. This is expected: these are reconstruction-only tokenizers with an 8×8 latent at 32×32 resolution, and the experiment is a controlled comparison among quantizers.

2. CelebA-HQ 40-attribute classification (AP)

We train a 40-way BCE linear probe on frozen MERVar-AE binary latents, using CelebA-HQ 256×256 (28K train / 2K test) and the official CelebA labels. The encoder is the CIFAR-10 model and has never seen a face.

RepresentationProbe input dimMean AP (%)
Chance level (mean positive rate)–24.9
MERVar-AE binary latent (10×64×64)4096064.3
MERVar-AE binary latent, 8×8 average-pooled64064.0

Per-attribute AP (%).

HighestMale 99.3No_Beard 98.0Wearing_Lipstick 97.2Eyeglasses 95.9
Smiling 94.8Young 94.0Mouth_Slightly_Open 92.2Heavy_Makeup 91.8
LowestBlurry 8.8 (4 positives)Wearing_Necklace 25.9Narrow_Eyes 28.5Straight_Hair 31.1

Reference: published results in the transfer setting. Because our encoder is not trained on CelebA, we compare with the FFHQ-trained setting of DiTi/DBAE Table 1, where the encoder is also trained on another dataset and then probed on CelebA attributes.

MethodRepresentationEncoder training dataProbe dataMean AP (%)
IB-GAN512-d continuousFFHQCelebA42.8
β-TCVAE512-d continuousFFHQCelebA43.2
PDAE512-d continuousFFHQCelebA59.7
Diff-AE512-d continuousFFHQCelebA60.5
SimCLR512-d continuousFFHQCelebA60.8
DiTi512-d continuousFFHQCelebA61.4
DBAE-d512-d continuousFFHQCelebA65.6
DBAE512-d continuousFFHQCelebA66.4
MERVar-AE (ours)10-bit binary codesCIFAR-10CelebA-HQ64.3 / 64.0 (640-d)

These results are listed as context, not as a direct comparison. The published methods use continuous 512-d encoders with diffusion decoders and the original CelebA split, and their FFHQ → CelebA transfer stays within faces.

Even so, an out-of-domain, reconstruction-only binary code reaches the same order of AP, including at 640-d.

Summary

The binary latents of MERVar-AE are useful beyond reconstruction:

  • On CIFAR-10, they give the best linear-probe accuracy among discrete quantizers at an equal 640-bit budget.
  • On CelebA-HQ, they reach 64.3 mean AP without any face-domain training.

Both experiments and the full probe details will be added to the revised paper.

AuthorsOpenReview ↗

Additional Comparison Experiments with Semantic Hashing

We provide three additional experiments to demonstrate the differences between the proposed MERVar-AE and semantic hashing [1]. They compare (1) encoder-output scale and effective decoder-input noise, (2) hard-code reconstruction and latent refinement, and (3) the 2011 hard-forward STE variant. Semantic hashing adds Gaussian noise before sigmoid, cSH=sigmoid⁡(a+4ϵ)c_{\mathrm{SH}}=\operatorname{sigmoid}(a+4\epsilon), whereas MERVar-AE adds noise after tanh, z~=tanh⁡(a)+0.3ϵ\tilde z=\tanh(a)+0.3\epsilon. The results show that these different computations yield different latent perturbations and optimization behavior, even when initial reconstruction scores are close.

1. Pre-sigmoid noise is associated with larger encoder outputs and a smaller effective perturbation. On all 10,000 CIFAR-10 test images, semantic hashing's median ∣a∣|a| is 16.858, versus 3.679 for MERVar-AE. Although aa is unbounded in both methods, semantic hashing learns much larger logits, helping its sigmoid codes stay near 0/10/1 under pre-activation noise.

Pre-sigmoid σ=4\sigma=4 does not imply a perturbation of magnitude 44 at the decoder. In the common [−1,1][-1,1] decoder-input coordinates, the measured mean absolute perturbation is 0.084 for semantic hashing, versus 0.239 for MERVar-AE with post-tanh σ=0.3\sigma=0.3. Semantic hashing also has a much smaller local derivative, as shown below.

Full-test measurementSemantic hashingMERVar-AE
Median abs(a)16.8583.679
Median decoder-input derivative7.30×10−87.30\times10^{-8}2.55×10−32.55\times10^{-3}
Mean absolute perturbation at decoder input0.0840.239

Thus the semantic-hashing decoder receives a smaller effective perturbation, while saturation makes the logit-to-code mapping locally flatter.

2. The difference becomes clearer during subsequent latent optimization. On the full 10,000-image CIFAR-10 test set, hard-binary reconstruction PSNR is 25.666 dB for semantic hashing and 25.745 dB for MERVar-AE. The AE reconstruction difference is modest. We then freeze each entire model and optimize only the encoder-produced logits for the same 100 test images, using MSE, batch size 88, Adam learning rate 0.020.02, and 3,000 steps. Each method retains its own noise location during optimization; neither uses STE. We evaluate the actual hard binary codes without noise:

Frozen modelBefore optimizationAfter 3,000 stepsGain
Semantic hashing, noise before sigmoid25.61226.166+0.554 dB
MERVar-AE, noise after tanh25.62326.592+0.969 dB

The methods start only 0.011 dB apart. After equal steps, MERVar-AE is 0.426 dB higher, with a 0.415 dB larger gain. This is consistent with the measured perturbations: sigmoid saturation suppresses pre-activation noise, whereas post-tanh noise continues to perturb the decoder input. The 100-image numbers measure per-image refinement, not full-test generalization.

3. A 2011-style STE control degrades under longer latent optimization. The image-retrieval extension [2] rounds sigmoid codes to 0/10/1 in the forward pass and ignores rounding in the backward pass. We applied this hard-forward, sigmoid-backward rule to latent optimization of the same trained semantic-hashing network and the same 100 images, keeping the MSE loss, batch size, and learning rate unchanged. Its hard-code PSNR rises to 25.846 dB at 500 steps, but then falls to 25.345 dB at 3,000 steps. The pre-sigmoid-noise path without STE instead reaches 26.166 dB at 3,000 steps. This is a latent-optimization control rather than a reproduction of the 2011 training pipeline; the two paths also differ in their use of Gaussian noise.

Experimental setup. For this comparison, we use the same CIFAR-10 data, VQ-VAE architecture, 640-bit bottleneck, and hard-code PSNR evaluation. Semantic hashing starts from the MERVar-AE demo checkpoint and receives 50,000 additional MSE updates with pre-sigmoid noise. For latent refinement, both frozen models use the same 100 images, MSE, batch size 8, Adam learning rate 0.02, and 3,000 steps.

Overall, although the proposed MERVar-AE and semantic hashing both use noise to encourage binarization, they differ in mathematical foundation, noise placement, and the latent representations they learn. Under the shared architecture and latent-refinement protocol, these differences are most evident in MERVar-AE's larger latent-optimization gain.

[1] Semantic Hashing, IJAR 2009.

[2] Using Very Deep Autoencoders for Content-Based Image Retrieval, ESANN 2011.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.