- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies image codes with entries entries. An encoder output is passed through tanh, so each coordinate lies in , then the Gaussian noise is added, and the encoder and decoder are trained to reconstruct the image. The claim is that the noise pushes different images apart, so coordinates move toward . Once enough coordinates are near , the noise is turned off. At this stage each coordinate is replaced by its sign, the encoder is frozen, while decoder not. Freezing the encoder avoids a gradient through the sign. There is theoretical argument on overlapping noise clouds. The paper also includes experimental validation of the method and comparison with existing approaches.
Critical strengths
The paper uses a well-known and effective idea, i.e., to inject noise so that the codes become nearly binary. Under a shared training setup, the experiments show that the method performs well.
Critical weaknesses
MERVar-AE method hardly can be claimed to be novel, as it follows the mechanism proposed in [1]. It has already been proposed to add to add Gaussian noise, for example, at a saturating code layer and minimize reconstruction error, so that intermediate activations become unreliable and the codes saturate (and then threshold). [2] use that method on images and report that rounding is as effective. The related-work paragraph of this manuscript cites [1] and presents MERVar as the mathematical account of the same phenomenon. The account does not add any new mechanism.
I’d like to note that the proposed overlap coefficient is basically one minus total variation, i.e., OC$(p, q) = 1 - (p, q)$. In this regard, the claims of paper become even clearer. In particular, Theorem 3.5 is the standard formula for the total variation of two equal isotropic Gaussians followed by the chain rule.
Table 3 separates the definition from the reported run. With tanh, the maximum overlap at 16K updates is about 0.91, so the codes are not -mutually exclusive for any . Removing tanh is what drives the maximum overlap to order . Thus, saturation and mutual exclusivity, as defined, seem to occur in different ablations.
Table 7 only ranks methods under a short shared budget (50K CIFAR-10 updates, then 10K updates on the target set). On ImageNet the second stage uses about 300K images. This is the whole comparison with VQ-VAE, FSQ, LFQ, BSQ, and the other listed methods. I think it is not evidence about those methods as large-scale tokenizers, which is the reading invited by "representative state-of-the-art quantization techniques".
Minor points. The factor in (9) comes from quadratic inequality for which is 2-strongly convex in the target . Remark A.1 does not say in which argument the loss is strongly convex, and strong convexity in the prediction does not yield (9). Independence of the noise is unused in both proofs and cannot affect a marginal overlap. Unimodality and symmetry are unused in Theorem 3.3, they belong only to the non-Gaussian claim in Remark 3.6, which is not proved. That remark cites Appendix A.3.3 as empirical support, however Table 11 in Appendix A.3.3 reports reconstruction PSNR, not distances or overlap coefficients.
[1] Salakhutdinov, Ruslan, and Geoffrey Hinton. "Semantic hashing." International Journal of Approximate Reasoning 50.7 (2009): 969-978.
[2] Krizhevsky, Alex, and Geoffrey E. Hinton. "Using very deep autoencoders for content-based image retrieval." ESANN. Vol. 1. 2011.
Questions for authors
-
In Table 3, the mean overlap falls to about 0.009, but the largest overlap stays near 0.91, and the definition 3.1 is a bound on the largest overlap. Which number do you use to show that training follows Theorems 3.3 and 3.5?
-
Table 7 uses 50K CIFAR-10 updates and then 10K updates on the target set. Could you add larger-scale experiments, closer to how VQ-VAE, FSQ, LFQ, and BSQ are usually trained, to support the comparison with representative state-of-the-art quantization methods?
3 comments
I have seen the responses given by the authors to my and other reviews. I appreciate their clarification. I have raised my score and I assume that the refinement of the manuscript will be conducted (including theoretical part).
Thank you for the discussion
Thank you for considering our responses and raising your rating. We will include the semantic hashing and large-scale tokenizer comparisons, clarify the theorems’ assumptions and their connection to reconstruction, and explain how the mean overlap in Table 3 differs from the maximum in Definition 3.1.
MERVar Theory, Overlap Statistics, and Large-Scale Validation
Thank you for recognizing the method's effectiveness under a shared training setup. We address the novelty, theoretical scope, and experimental-scale concerns below.
W1: Contributions Beyond the Shared Mechanism
The connection to semantic hashing is already discussed in Related Work. Our principal contribution is the formulation and mathematical analysis of MERVar, particularly the MSE Lower Bound Theorem (Theorem 3.3), which relates reconstruction error to distance-weighted overlap. The Mutual Exclusivity Theorem (Theorem 3.5) further characterizes Gaussian pairwise separation. Beyond this analysis, we develop and evaluate STE-free binary image tokenization for autoregressive generation, whereas semantic hashing targets retrieval and its 2011 image-retrieval extension uses an STE-type estimator. A shared mechanism alone does not negate these theoretical, optimization, and empirical contributions, supported by our controlled experiments, downstream generation results (Table 14), and new large-scale comparisons.
W2: OC–TV Equivalence and the Role of Theorem 3.5
The identity OC(p,q)=1−TV(p,q) is an equivalent representation, rather than an alternative to our analysis. We use OC because it directly quantifies the overlap responsible for reconstruction ambiguity and yields a natural expression of the MSE lower bound. It also makes the definition of ε-MERVar transparent: OC(pi,pj)≤ε directly bounds pairwise overlap, equivalently TV(pi,pj)≥1−ε. Rewriting the bound using 1−TV leaves its content unchanged and does not establish the bound itself.
The Mutual Exclusivity Theorem (Theorem 3.5) uses the Gaussian overlap formula and the chain rule to characterize the separation direction induced by overlap minimization. Its role is to make this optimization property explicit within the MERVar framework. This observation does not subsume the MSE Lower Bound Theorem (Theorem 3.3), which establishes the relationship between reconstruction error, distributional overlap, and pairwise target distances. The known OC–TV identity and Gaussian formula therefore do not negate this additional theoretical contribution.
W3 / Q1: Separation Dynamics and Overlap Statistics
Table 3 demonstrates the progressive separation of latent distributions during training, rather than claiming that every pair has reached a small overlap. With tanh, the mean OC decreases from approximately 0.99 to 0.009, while the maximum OC of approximately 0.91 reports the remaining worst-case overlap. These statistics describe the separation dynamics; they are not used to claim uniform satisfaction of Definition 3.1 for a small ε.
Figure 4 directly evaluates the distance-weighted overlap lower bound in Theorem 3.3 against reconstruction MSE; neither the unweighted mean nor the maximum alone represents that bound. Theorem 3.5 characterizes the separation direction under pairwise Gaussian overlap minimization, without guaranteeing uniformly small overlap for all pairs during reconstruction training. The no-tanh ablation in Table 3 further examines how removing the bounded latent constraint affects separation.
W4 / Q2: New Large-Scale Tokenizer Comparisons
Table 7 evaluates methods under identical encoder–decoder architectures and a shared training budget; it isolates behavior under controlled conditions rather than reproducing each method's largest configuration. Our new large-scale results address the separate question of competitiveness with large tokenizers.
Minor Points: Proof Assumptions and Non-Gaussian Scope
Strong convexity. The relevant argument in Remark A.1 is the target: for fixed prediction y, v↦∥y−v∥2 is 2-strongly convex. Equivalently, Eq. (9) follows directly from
∥y−xi∥2+∥y−xj∥2=2y−2xi+xj2+21∥xi−xj∥2.
We will specify the argument explicitly; no convexity of the neural network is required.
Noise assumptions. Theorem 3.3 uses sufficient assumptions consistent with our model, although its proof only requires marginal densities. The Gaussian assumption in Theorem 3.5 enables an explicit overlap formula and separation direction. We will clarify these roles.
Non-Gaussian behavior. Table 11 evaluates the applicability of MERVar-AE across different noise distributions through reconstruction performance. It is not intended as a proof of non-Gaussian separation. Theorem 3.5 concerns isotropic Gaussian noise; the additional experiments demonstrate the method’s empirical applicability beyond this setting. We will clarify this distinction in Remark 3.6.