Recover Semantics, Not Codewords: An Asymmetric Paradigm for Vector-Quantized JSCC

Vector-quantized semantic communication commonly follows a recover-then-reconstruct paradigm: the receiver detects a transmitted symbol, retrieves its source codeword, and then reconstructs the image. This design resolves channel ambiguity through a discrete decision, although a codeword error need not imply poor reconstruction. We argue that faithful source reconstruction does not require exact codeword recovery. Our insight is to jointly learn which source representations become neighbors in the channel and how observations within those neighborhoods are reconstructed. We introduce AsymVQ-JSCC, which retains a discrete source vocabulary while coupling source-aware irregular QAM with continuous semantic recovery. The transmitter learns constellation coordinates from source relationships and refines their geometry through the image reconstruction objective. Building on this geometry, a continuous receiver maps raw observations into semantic latents, learning denoising jointly with the decoder. Bypassing hard detection and codebook lookup allows observations between constellation points to support reconstruction through latents beyond the source vocabulary. Each spatial token occupies one channel use, independently of vocabulary size. Across four image datasets, bandwidth ratios, and SNRs, AsymVQ-JSCC achieves competitive PSNR and perceptual quality with graceful degradation under both AWGN and Rayleigh fading. In a fixed-checkpoint intervention on Kodak, projecting receiver latents to the nearest source codeword, while leaving the transmitted symbols, channel observations, and model weights unchanged, reduces PSNR by 1.46–2.57 dB and increases LPIPS by 0.063–0.077 across the tested SNRs. These results provide direct evidence that reconstruction benefits from continuous receiver representations beyond the source vocabulary.