DLL: Deep Latent Learning via Layer-Wise Latent Supervision

Traditional autoregressive predictive learning paradigms rely primarily on surface-level observable objectives, such as next discrete tokens or future continuous values. This paper argues that observable samples induce additional targets in the deep latent space, namely the contextualized representations embedded in deep layers. We propose $\text{Deep Latent Learning (DLL)}$. DLL trains the model to simultaneously predict what the next observable target is and how that target should be represented in the deep latent space. We emphasize the necessity of supervising deep layer-wise representations, as deep layers leverage sequence-wide context through attention mechanisms far more effectively than shallow layers. To complement deep latent supervision, DLL introduces a self-purification module that anchors latent targets to the observables to prevent representation collapse. DLL yields robust improvements on established benchmarks across three domains, such as ImageNet-1K in computer vision, TableBench in natural language processing, and GIFT-Eval in time series forecasting.