- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Cola2, a hierarchical language modeling framework that combines dynamic semantic compression, balanced discrete generation, and continuous–discrete co-adaptation. The framework uses generation-oriented supervision to learn predictive latent representations, a block-causal diffusion decoder trained with varying latent reliability, and detached Flow Matching for latent prior learning. Experiments across five downstream tasks and approximately 210B training tokens demonstrate improvements over the original Cola DLM and competitive performance against autoregressive and diffusion-based baselines.
Critical strengths
-
The framework addresses the interaction between latent compression and discrete generation in a systematic manner, with practically motivated designs for representation learning and decoder robustness.
-
The paper provides extensive component analyses. The experiments offer useful insights into hierarchical language generation.
Critical weaknesses
-
A central motivation of Cola2 is to reduce the sequence length through continuous semantic compression. However, the scaling experiments show that the uncompressed Cola-MDM generally outperforms Cola2-MDM. Without a clear accuracy–efficiency comparison, it remains difficult to assess whether the proposed compression offers a meaningful advantage over its uncompressed counterpart.
-
The dynamic segmentation mechanism identifies semantic boundaries primarily through cosine differences between adjacent token embeddings. However, such local representation differences do not necessarily correspond to meaningful semantic boundaries. Although the proposed segmentation outperforms fixed-length and random segmentation, these results establish the benefit of adaptive partitioning rather than demonstrating that the learned boundaries reflect semantic structure. Moreover, the information-theoretic formulation introduces a high-level semantic variable S, but its relationship with the learned latents is not directly evaluated. Additional analysis of the learned span boundaries, or comparisons with alternative semantic segmentation methods, would help substantiate the claim of semantic compression.
Questions for authors
See Weaknesses.
2 comments
Dear Reviewer hL13,
Thank you very much for your kind response and for taking the time to reconsider our rebuttal and revisions. We sincerely appreciate your acknowledgment that most of your concerns have been addressed.
Thank you once again for your valuable time and consideration!
Sincerely, The Authors of Submission #7368
Thanks to the response. Most of my concerns have been addressed. I will retain my current score.