acceptodds
Under review as a conference paper at ICLR 2027

RST: Routes for Certifying and Constructing Prescribed Information in Variational Autoencoders

Abstract

Posterior-collapse diagnostics can establish that a VAE uses its latent representation, but they do not determine whether a specified information view crosses a representation/readout interface fixed before training. Reconstruction-Student-Teacher (RST) makes that stronger requirement explicit: R denotes Reconstruction, S the fixed Student witness, and T the frozen Teacher specification. Let be a full-support categorical Teacher, , , and let an audited representation satisfy . With , , and , we prove and for every input-independent , with equality attained by the centered RST witness. Thus certifies transmission of the declared Teacher information through the specified interface. In the main experiments is the deterministic encoder mean, while reconstruction uses . A centered regular-simplex witness turns the certificate into an explicit construction: a minimum-dimensional closed-form Teacher code, its complete affine solution fiber, exact witness-visible/witness-null routing, a margin-energy path, strict function-space departure from the centered input-independent point, certified preservation cylinders, and orthogonal multi-view composition. Five-dataset experiments test boundary crossing, stronger KL pressure, Teacher counterfactuals, witness-null routing, escape, and preservation. On CIFAR-100, learned-head auxiliary (Aux) codes are probe-readable while their predeclared margins are negative. Cross-seed Aux-head reuse falls from native accuracy to , whereas one fixed RST witness directly reads all five Conv-RST encoders at . The same witness is also certificate-positive on five independently optimized ResVAE RST encoders, with coarse/fine margins and ; ConvRes learned-head reuse is near chance, while a ridge-affine adapter makes all transferred margins positive. These tests validate a functional consequence of prescribed alignment; they do not claim fixed simplex classifiers or compatible representations as new.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
variational autoencoders, posterior collapse, representation learning, information theory, prescribed alignment, simplex geometry, representation compatibility

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zdL7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RST, a framework for certifying and constructing prescribed information in variational autoencoders (VAEs). Given a frozen soft-label teacher and a pre-declared, fixed student witness, the paper defines a margin (G_T = I_T - L_{TS}) and uses a positive margin to certify a lower bound on the information retained in the latent representation. It further establishes a boundary result for input-independent encoders and develops constructive results based on regular-simplex geometry, including routing, movement, preservation, and composition.

The empirical evaluation consists of experiments on five datasets using PCA–GMM teachers, as well as controlled experiments on CIFAR-100 with coarse- and fine-label teachers. The paper also evaluates the reuse of a fixed witness across random seeds and VAE architectures without retraining. The central claim is that RST provides a certificate tied to a pre-specified representation-and-readout interface, rather than merely demonstrating that information is decodable by a post-hoc probe.

Critical strengths

A clearly specified certification target with constructive theoretical results. The paper formalizes a lower bound on information retained through a pre-declared, fixed readout interface and derives associated boundary and construction results. This provides a mathematically explicit framework for studying representation properties under a specified interface, rather than relying exclusively on empirical probing.

Controlled evaluation of fixed-witness reuse. The experiments include comparisons with auxiliary-loss controls and evaluate witness reuse across seeds and architectures without retraining. These experiments help examine whether the proposed certificate captures properties of the specified interface that are not reflected by a generic post-hoc readout alone.

Critical weaknesses

The practical and diagnostic value of the certificate remains insufficiently established. Since the proposed margin is directly related to the training objective, a positive margin on models trained with the corresponding auxiliary loss may primarily reflect successful optimization of that objective. The auxiliary-loss control and cross-architecture, no-refit experiments provide useful evidence for the proposed interface-specific interpretation, but it remains unclear what meaningful failure modes the certificate rules out beyond these demonstrations. The paper would be stronger if it established a clearer distinction between the proposed certificate and existing decodability or probing-based evaluations.

The semantic scope and practical relevance of the demonstrated information views are limited. The broader experiments use PCA–GMM teachers, while the CIFAR-100 experiments use dataset-defined labels. These settings demonstrate the framework on clustering-derived or predefined classification targets, but do not establish whether it can certify information about external semantic attributes or variables relevant to downstream tasks. The paper also evaluates multiple variants, including prefit, routing, joint, and auxiliary-loss configurations, making it difficult to identify which variant supports the principal claims and whether the reported conclusions hold consistently across variants.

Questions for authors

Diagnostic value of the certificate. Beyond the auxiliary-loss control and cross-architecture, no-refit experiments, can the authors identify a concrete failure mode that is ruled out by (G_T > 0) but would not be ruled out by conventional probing or decodability evaluations? What additional diagnostic value does the proposed certificate provide?

Scope of semantic certification. Can the framework certify information about a semantic attribute or downstream-relevant variable that is neither a clustering assignment nor a dataset-provided label? If such cases are outside the scope of RST, what assumptions define the framework's applicability, and how should its intended use be interpreted?

Consistency across method variants. Which single method variant is responsible for the paper's main claims and headline results? Are the theoretical guarantees and empirical conclusions consistent across the proposed variants, and which variant should practitioners use as the default?

Reviewer_4a4UOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a prescribed alignment problem that goes beyond whether the latent representation of a Variational Autoencoder (VAE) is simply input-dependent or whether specific information can be decoded via a post-hoc probe. It ensures that Teacher information specified prior to training can be directly read through a fixed readout/interface, also specified before training.

In order to achieve this goal, the authors define a Reconstruction–Student–Teacher (RST) architecture. The Teacher TT provides a desired categorical information view TXT_X for an input XX, while the Student SS is a predeclared fixed witness/readout that accepts only the latent representation as input. In the main experiments, the representation evaluated by the certificate is not a sampled latent but the encoded input (posterior mean) Z=μϕ(X)Z=\mu_\phi(X).

The authors define

L_{TS}=\mathbb{E}\,\mathrm{KL}(T_X\Vert S(\cdot\mid Z)), \qquad G_T=I_T-L_{TS} ,$$ where $I_T$ is the total amount of information the Teacher has about the input $X$, and $L_{TS}$ is the average mismatch that indicates how poorly the current latent representation $Z$ can predict the Teacher's output when interpreted as a fixed Student/witness $S$. Then it is shown that $G_T \le I(C;Z)$. Therefore, if $G_T>0$, it certifies that the specified Teacher information exists on that route, and every input-independent representation satisfies $L_{TS}\ge I_T$. The authors select a regular-simplex linear-softmax witness to derive the closed-form code $u_T(x)$ that exactly realizes a $K$-class full-support Teacher in a $K-1$-dimensional visible subspace, along with the entire solution fiber $u_T+\ker V$, witness-null invariance, the analytic margin–energy path, the local function-space escape direction, the certified preservation cylinder, and orthogonal multi-view composition. Experiments demonstrate that RST achieves a positive certificate margin across multiple image datasets, and on CIFAR-100, it places coarse and fine information in distinct simplex blocks. Furthermore, while learned auxiliary heads can read Teacher information from new probes but fail with predeclared witnesses, RST representations allow the same fixed witness to be reused across multiple seeds and ConvVAE/ResVAE architectures.
Critical strengths
  1. The distinction in the problem formulation itself is clear and useful. The paper clearly distinguishes between latent activity, post-hoc decodability, and prescribed alignment. “Information exists within the representation” and “a pre-specified consumer can read that information without refitting” are different requirements, and the rotation counterexample and Aux controls illustrate this difference intuitively. This distinction is particularly meaningful in applications where a coordinate contract is actually required, such as model replacement or modular interfaces.

  2. The core certificate is simple but correct. The result GT>0  ⟹  I(C;Z)>0G_T > 0 \implies I(C;Z) > 0 is a clear sufficient condition, and the input-independent baseline is defined in a way that makes it attainable. The authors also consistently specify that they do not conclude collapse for GT≤0G_T \le 0. The claim boundary of the theory is relatively well-managed.

  3. Simplex geometry is used to neatly express various properties within a single framework. The fact that the minimum-norm inverse, affine fiber, visible/null decomposition, analytic path, preservation radius, and orthogonal composition are all derived from the same linear geometry makes for a clean exposition. In particular, the exact fiber uT+ker⁡Vu_T+\ker V and witness-null invariance clarify the interpretation of the interface.

  4. The authors impose some self-imposed limits on excessive claims. The paper clearly states that the certificate do not claim to solve semantic completeness or universal collapse. Furthermore, it does not claim that the analytic path is optimizer-independent.

  5. The experimental controls are well-suited for demonstrating coordinate mismatch. The result that the aux representation is readable as a post-hoc probe but not as a fixed witness, and that adding an adapter restores cross-seed/cross-architecture transfer, is effective in distinguishing between “information loss” and “interface mismatch.”

Critical weaknesses
  1. In my opinion, the biggest problem is that the novelty is limited. Nearly all of the paper’s core components are already well-known. Aligning backbone representations to a predefined coordinate system using a fixed classifier has been directly studied since Hoffer et al. (2018), and regular simplex/polytope fixed classifiers were analyzed in Pernici et al. (2022) and Yang et al. (2022). Furthermore, Biondi et al. (2026) obtain stationary representations using a d-simplex fixed classifier and theoretically link this to representation compatibility. This is very close to the motivation for cross-seed/cross-architecture fixed-reader reuse in this paper. The substantive new element of RST appears to be that it “combines fixed/simplex readout with a Teacher-relative mutual-information lower-bound certificate.” However, as the authors themselves acknowledge, the certificate itself is based on the existing variational MI bound and the earlier Teacher-guided constant-collapse certificate. The paper also states that the input-independent boundary itself is not a new result. Consequently, the contribution of this paper appears to be a combination of several existing results within a single notation and interface framework, rather then a new representation-learning principle.

  2. RST training is essentially very similar to fixed-head auxiliary supervision, and the comparison with this baseline is insufficient. The actual soft objective is

\lambda_{TS}\mathrm{KL}(T_X\Vert S(\mu_\phi(X))) .$$ In other words, the encoder features are aligned with a linear-softmax reader fixed prior to training. This is essentially identical to the core mechanism in the fixed classifier literature. Although the paper compares its approach to a learned auxiliary head, more direct baselines for evaluating novelty would include a generic fixed classifier, a random fixed orthogonal classifier, a fixed ETF/simplex classifier, or a stationary-representation method. The current experiments primarily reaffirm the obvious difference that “the learned head varies in coordinates, whereas the fixed head keeps coordinates fixed.” 3. The use of the simplex witness is also very similar to existing mixture/prototype geometries. The regular simplex is a classical geometry that symmetrically arranges $K$ classes, and the authors do not claim this to be novel. Furthermore, the simplex-softmax witness has the same form as the responsibility map of a spherical $K$-component GMM with the same covariance and simplex means. Therefore, the categorical/mixture structure itself is not new. Mixture-prior VAEs, such as VaDE, already incorporate a GMM structure into the generative latent model of the VAE. The difference with RST is that it uses a GMM-like map as a fixed readout instead of changing the prior. While this difference is clear, it is questionable whether it constitutes sufficient conceptual novelty. 4. The terms “canonical” and “minimal” make sense only within a very limited class. The $K-1$ minimality holds under the condition that the open set of all full-support $K$-categorical distributions can be represented by a linear-softmax witness. The canonicality of the simplex is also a result derived under a specific criterion, namely minimizing the spectral norm among standardized, minimal-dimensional linear witnesses. If the actual teacher distribution lies on a low-dimensional manifold or allows for nonlinear readouts, this minimality does not hold. 5. The “escape” result is not a guarantee for actual VAE optimization. What the paper demonstrates is that the right derivative of the alignment loss is negative in the manually chosen function-space direction $u_\alpha=\alpha u_T$. However, whether SGD can achieve this direction under a neural encoder parameterization depends on the encoder Jacobian and the ELBO gradient. The authors also explicitly state that they do not claim optimizer-independent convergence because there is no uniform lower bound on the Jacobian. Therefore, the phrase “escape from collapse” sounds stronger than an actual optimization guarantee. 6. The preservation results also pertain to conditional stability after an informative anchor has already been provided. The “Certified Cylinder” is a Lipschitz argument based on the assumption that a positive-margin representation was obtained first; it states that the margin is preserved if visible displacement is restricted to a sufficiently small value. While this is a useful safety statement, it does not constitute a result showing that an informative representation is generated or maintained during unconstrained training. 7. The certificate does not directly address the generative behavior of the VAE. The main certificate applies to the deterministic posterior mean $\mu_\phi(x)$, but the decoder uses the sampled $z\sim q_\phi(z\mid x)$. Therefore, a positive $G_T$ does not guarantee that the decoder actually uses the protected route, Teacher information is preserved in the sampled latent, or the aggregate posterior or generation quality improves. Because of this, the paper’s framing of posterior collapse sounds broader than the actual scope of the theorem. 8. The experimental evaluation metric is too directly linked to the training objective. RST is trained to minimize $L_{TS}$, and success is evaluated using $G_T = I_T - L_{TS}$. Therefore, an encoder with sufficient capacity obtaining a positive certificate is, to some extent, a self-fulfilling prophecy. In contrast, baselines such as VAE and InfoVAE are not trained to align with a pre-specified witness coordinate system; thus, obtaining a negative margin on a fixed witness is more likely due to a coordinate mismatch than a representation failure. The paper itself acknowledges that a negative margin is inconclusive. 9. When interpreting the cross-seed/cross-architecture results, one must also consider the fact that all RST models were trained to share a fixed interface. Since all RST models optimize loss with respect to the same Teacher and the same fixed witness, the fact that the same witness works across multiple models is an expected result given the design of the method. While this serves as a useful example of compatible representations, a more direct comparison is needed to demonstrate a new capability compared to existing literature on stationary/fixed-classifier compatibility. Overall, I do not believe there are any obvious errors in the proof. My concern is rather that the proven results are quite narrow, rely heavily on standard KL decomposition and simplex/tight-frame linear algebra, and that the more important claims regarding actual learning and VAEs are limited to a small set of experimental results.
Questions for authors
  1. What is the fundamentally new learning capability that RST offers compared to existing fixed-classifier/fixed-ETF representation learning? Please explain the differences from Hoffer et al. (2018), Pernici et al. (2022), Yang et al. (2022), and especially Biondi et al. (2024). which links d-simplex stationary representations to compatibility, in a way that goes beyond simply stating that “it was applied to a VAE” or “it has a certificate.” Can you include direct empirical baselines for comparison with these works?

  2. Why is the learned Aux head the primary comparison target? If RST’s central design choice is a predeclared fixed witness, shouldn’t the most direct baseline be a method that uses the same Teacher supervision but employs an arbitrary fixed linear head, a random orthogonal head, a fixed ETF classifier, or an existing stationary-representation objective?

  3. As previously noted, the simplex witness can be interpreted identically to a symmetric spherical-GMM responsibility map. In that case, what are the practical advantages offered by the RST formulation compared to existing categorical/GMM-structured VAEs or fixed-prototype latent models? In particular, how does it differ from existing approaches that predefine component semantics and align the responsibility loss with the Teacher target?

  4. The main theorem and experiments primarily certify μϕ(X)\mu_\phi(X), but the decoder uses sampled zz. Why aren’t sampled-latent certificates for S(z)S(z) or I(C;z)I(C;z) analyzed as key results? What exactly does the mean-route certificate guarantee regarding the behavior of the generative model?

  5. If “escape” does not guarantee parameter-space convergence, what additional insights does this result provide to the existing theory of posterior-collapse optimization? Since general auxiliary supervised losses can also generate nonzero feature gradients in the collapsed representation, what distinguishes RST?

  6. The limitation that all cross-architecture RST models use the same witness in their training objective has already been pointed out. Can “no-refit” transfer be compared to existing compatible/stationary representation methods under the same protocol?

  7. What is the relationship in the paper between being “certificate-positive” and downstream utility? Is there evidence that a larger GTG_T model consistently performs better in actual transfer, reconstruction, generation, robustness, or downstream tasks?

Reviewer_4eokOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a technique for testing the latent representation of a VAE to see if contains a set of qualities prescribed by a frozen teacher. The teacher provides a (set of) categorical labels. The test takes the form of a certification which measures whether a ‘student’ can learn the teacher specification from a VAE latent layer. The certificate margin is the difference between the teacher information and a student-based loss function.

Critical strengths

The paper addresses an open question – the interpretability of VAE representations, and the attendant problem of diagnosing posterior collapse. It discusses some geometric properties of the VAE latent space. It claims geometric rigour.

Critical weaknesses

The major weakness of the work is its lack of clarity. This stems from the complexity of the explanations. The abstract contains a 50 word sentence which, in spite of careful reading, remains very difficult to understand. It seems heavily based on ideas from a single paper (A Testable Certificate or Constant Collapse in Teacher Guided VAE (Zhang et al.2026).

Its contributions relate to the geometric properties of the projection of a set of data into a latent space, but it uses language unhelpful language and omits more terms commonly used to discuss such geometry (e.g. the word manifold, often used to describe multi-dimensional projections, only appears once in an appendix.)

These difficulties meant that I was unable to follow the technical theorems and corollaries or comment on their relevance to the overall objective.

Minor errors include referencing figures out of order (the first figure to be referenced is Figure 3), and weighting terms (beta_KL and lambda_TS) are referenced without being introduced.

Questions for authors

Could the language used in the paper be simplified?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.