- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The submission examines the connection between discrete codebooks that partition the latent space of a VAE. Under what conditions can we know if two codebooks agree? A worked counterexample demonstrates that standard metrics reported during training (such as per example KL or ELBO) can match while codebooks differ. The paper then establishes bounds on agreements of codebooks under different sampling measures (eg agg posterior and prior) to these standard metrics -- the KL and ELBO gap. Experiments on MNIST and SVHN lend support to the bounds.
Critical strengths
None
Critical weaknesses
The submission is poorly motivated and poorly written.
It comes as no surprise that codebooks on top of a VAE's latent space -- codebooks with no relation to training, nor specifics about their definition that might serve to link them in any way -- can disagree while the VAE functions perfectly. With a careful construction and a more subtle argument, there could be a successful hook for this work, but instead it's delivered in dramatic flair characteristic of AI-packaging. In the abstract: "Shannon (1948) took such a shared codebook as given; inside a VAE nothing enforces it." Well, yes, the codebook has only as much of a connection to the VAE as is integrated into its construction, and the only details about codebook creation before the empirics is "In applications, is fit on encoder statistics and is a nearest-prototype rule in the decoder's Bregman output geometry." (L167). Loosely, I cluster z and I cluster x, and then compare both in z by the preimage of the latter. Without some sort of conditioning on the clustering schema, I would never expect guarantees on the agreement of the clustering assignments besides what might arise downstream of (dis)agreements of the sampling distributions, which are connected to the VAE optimization.
The prose and figures are typical of AI outputs -- a sales pitch that impedes a close reading.
Questions for authors
Can the authors propose specific scenarios where the codebooks are constructed jointly with the express purpose of agreement, and demonstrate limits exist?
Reply to Reviewer DtHP, updated for the revised PDF
The construction, which the submitted main text did not state. You asked for scenarios in which the codebooks are constructed jointly, and for limits. In our VAE experiments the two maps are linked, and Section 2 of the revision now says how: centers μ_k are fitted to encoder means; the encoder-side code of a latent is its nearest center (or most probable mixture component); the output prototypes are the decoded centers g(μ_k); the decoder-side code is the nearest prototype of the decoded latent. Both codes carry the index of the center. Before the experiments, the submitted main text said only that the maps are taken as given, that c_enc is fit on encoder statistics and that c_dec is a nearest-prototype rule (the sentence you quote); neither it nor Section 6 said that the prototypes are the decoded centers or that both codes carry the center's index. That omission was ours.
Jointly constructed codebooks (new Theorem 1(III)). Shared prototypes are a joint construction, though not one built for agreement. A construction built for agreement, c_enc := c_dec, has agreement one for every decoder; its limit is a cost (one decoder pass per written latent), not a failure of agreement. For the shared-prototype construction of our experiments there is a limit. Let both maps be rebuilt from each model by this rule in its k-means form (globally optimal centers; the output side read in squared distance or in Bernoulli KL, the two divergences of our experiments). Then there are two models with the same data law and prior, perfect reconstruction, exact inference and a generative law equal to the data law, with the same centers and the same encoder-side map, that match on items (m1)–(m6) of Definition 1 and have agreement 1−1/n and 1/n, for any odd n ≥ 3 (Appendix A.7, with an exact verification script). The second model exchanges, in each block of the latent space, the datum at the center with a datum of another class, so the decoded center, and with it the code read for the rest of the block, changes class. Sharing indexed prototypes therefore does not make agreement visible in the training report. Agreement does hold for every decoder if the written code is itself read through the decoder, as above, or if only the centers themselves are decoded, as in a VQ-VAE (Appendix A.7, remarks (a) and (b)). The decoders of the pair are piecewise constant, the statement is for the k-means rule, and we do not claim that training produces such pairs.
On "no surprise". For Theorem 1(I) your reading is right, and the revision says so. Its two maps are fixed and do not depend on the model, and one of them is not a readout of the decoder's output (Appendix A.1 and A.3, Table 2). We also state that its pair is not well specified (the generative law puts mass 1/K on the data) and that some misfit is necessary: by the new Corollary 1, a well-specified model with exact inference has the same agreement on encoded data and under the prior, so two such models with a common prior and common fixed maps cannot differ in agreement. Part (III) is the statement for maps that are linked to each other and to the model, and its pair is well specified.
On guarantees. We agree that the guarantees are of the kind you describe. Theorem 2 bounds how far agreement can move between sampling laws by KL budgets, and the budget alone does not establish high agreement: a certificate also needs the measured agreement. The revision adds that, given only the measured agreement and the budget, the band cannot be narrowed (Section 4).
Presentation. We removed the Shannon sentence and the historical framing, rewrote the abstract, introduction and conclusion as plain statements, and removed claims that went beyond what is proved or measured ("no item … can reveal", "trained models behave as the theory predicts", "best previously available"). The prose was drafted and revised with LLM assistance and the figures are drawn by scripts written with it; the AI-use statement, which we have made more specific, discloses both.
We would welcome your view on whether Part (III) answers your question, and on what else you would need to see.