- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes JPEG-DLM, a continuous diffusion language model that jointly learns compressed latent representations, a flow matching model, and a token decoding module. Its joint-embedding prediction (JEP) objective predicts clean embeddings produced by an EMA target compressor from corrupted features. The JEP predictor shares network parameters with the flow matching model, while feature reconstruction and token-level cross-entropy supervise decoding. Experiments on LM1B and OpenWebText demonstrate improved generation throughput and lower generative perplexity.
Critical strengths
-
Reducing the latent sequence length lowers the computational cost of repeated model evaluations. The reported throughput improvement over ELF is substantial and supports the practical value of latent compression.
-
On LM1B, JPEG-DLM achieves substantially lower Gen-PPL than the evaluated baselines while maintaining competitive MAUVE and improving throughput.
-
The paper examines compression rates, latent widths, network sharing, loss components, corruption strengths, and embedding regularization. The appendix also acknowledges limitations in long-sequence generation and discusses potential evaluation confounds.
Critical weaknesses
-
Joint learning of latent representations and generative models already exists, as acknowledged in the paper. Table 2 shows a considerably larger performance degradation from removing network sharing than from removing JEP. Since network sharing follows ELF, the experiments do not fully establish what the proposed JEP formulation contributes beyond existing shared-network training and noisy reconstruction objectives.
-
The comparison does not isolate the benefit of joint training. JPEG-DLM and COSMOS differ in several aspects beyond their joint versus two-stage training schedules. Moreover, the main comparison retains final-layer T5 features for COSMOS despite the appendix demonstrating substantial improvements from earlier-layer features. A controlled two-stage version of JPEG-DLM and comparisons with existing joint-training methods would provide stronger evidence.
-
The latent-space analyses support narrower conclusions than claimed. Higher correlation between latent positions does not necessarily imply a distribution that is easier to model; it can also reflect redundancy. Similarly, low Gen-PPL along latent interpolations demonstrates fluent decoding but does not establish preservation of the underlying information. These analyses do not sufficiently distinguish robust representations from decoding that favors generic outputs.
Questions for authors
-
Could the authors compare JPEG-DLM with a two-stage variant using the same architecture, comparable training objectives, and a matched total training budget to isolate the benefit of joint training? How does JPEG-DLM compare with existing joint-training approaches such as LDLM and Cola DLM?
-
What additional benefit does EMA-target JEP provide over ordinary noisy autoencoding or consistency learning? Could the authors provide controlled ablations that separately remove the EMA target mechanism and feature corruption?
RE: Rebuttal
Thank you for the detailed responses and additional experiments. The new comparisons and ablations substantially address my concerns. Please include the training-budget details for the two-stage comparison and qualify the conclusions drawn from the latent-space analyses in the revised manuscript.