Beyond the Encoder: Latent Representation Learning via Parametric Soft Discrete Random Variables

We propose the Parametric Soft Discrete Random Variable Auto-Decoder (PSDR-AD), a novel encoder-free framework for learning latent representations through directly optimized Parametric Soft Discrete Random Variables (PSDR-Vars). Our approach is motivated by the hypothesis that encoder-produced representations may not be optimal for image reconstruction because they are restricted by the learned encoder mapping. Specifically, each image is associated with a dedicated PSDR-Var, a directly optimizable discrete latent variable with a soft stochastic relaxation induced by noise injection. During training, the PSDR-Vars are fed into a shared decoder to reconstruct their corresponding images, while the PSDR-Vars and decoder are jointly optimized. During testing, the decoder parameters are fixed, while newly initialized PSDR-Vars are assigned to unseen images and optimized to minimize their reconstruction errors. Unlike conventional autoencoders, which treat latent extraction as a forward function evaluation that maps an input image to the output of an encoder, PSDR-AD treats it as an inverse optimization problem: given a target image as the desired output of the decoder, it optimizes the decoder input to reproduce that output. This removes the additional constraint imposed by the encoder mapping and can improve reconstruction quality even when PSDR-AD is applied to a decoder from a pre-trained autoencoder. Comprehensive experiments demonstrate the effectiveness of PSDR-AD for latent representation learning. Using the same pre-trained FLUX-VAE decoder, PSDR-AD reduces the rFID from 0.176 to 0.005 on ImageNet through direct latent optimization, achieving a lower rFID than existing continuous reconstruction methods.