acceptodds
Under review as a conference paper at ICLR 2027

Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation

Abstract

Hierarchical language generation separates continuous semantic modeling from discrete text realization, but its effectiveness depends on what information the latent space should compress and how the discrete generator should use it. We propose Cola2, which jointly studies continuous semantic compression, the discrete reconstruction–generation balance, and their co-adaptation. Cola2 dynamically segments text into semantic spans, controls the compression rate with an auxiliary-loss-free negative-feedback mechanism, and combines reconstruction and generation supervision to learn compact and predictive latent representations. A block-causal masked diffusion model (MDM) decoder is jointly trained under clean, noisy, and dropped latent conditions, and classifier-free guidance (CFG) adjusts the balance between semantic reconstruction and autonomous generation. During training, token-prediction gradients update the encoder through the latents, driving the continuous representation and discrete generation to co-adapt, while Flow Matching prior learning is decoupled from representation learning via stop-gradient. At inference, the generated text is re-encoded under the fixed span layout, keeping the latent history consistent with the actual text context. Systematic component analyses validate the core designs, and on five downstream tasks with about 210B training tokens, Cola2 exhibits competitive scaling potential, providing empirical support for hierarchical language generation with continuous–discrete co-adaptation.

Primary area:
generative models
Keywords:
Diffusion Language Model, Continuous Latent Language Modeling, Semantic Compression, Hierarchical Language Generation, Latent Diffusion, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hL13OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Cola2, a hierarchical language modeling framework that combines dynamic semantic compression, balanced discrete generation, and continuous–discrete co-adaptation. The framework uses generation-oriented supervision to learn predictive latent representations, a block-causal diffusion decoder trained with varying latent reliability, and detached Flow Matching for latent prior learning. Experiments across five downstream tasks and approximately 210B training tokens demonstrate improvements over the original Cola DLM and competitive performance against autoregressive and diffusion-based baselines.

Critical strengths
  1. The framework addresses the interaction between latent compression and discrete generation in a systematic manner, with practically motivated designs for representation learning and decoder robustness.

  2. The paper provides extensive component analyses. The experiments offer useful insights into hierarchical language generation.

Critical weaknesses
  1. A central motivation of Cola2 is to reduce the sequence length through continuous semantic compression. However, the scaling experiments show that the uncompressed Cola-MDM generally outperforms Cola2-MDM. Without a clear accuracy–efficiency comparison, it remains difficult to assess whether the proposed compression offers a meaningful advantage over its uncompressed counterpart.

  2. The dynamic segmentation mechanism identifies semantic boundaries primarily through cosine differences between adjacent token embeddings. However, such local representation differences do not necessarily correspond to meaningful semantic boundaries. Although the proposed segmentation outperforms fixed-length and random segmentation, these results establish the benefit of adaptive partitioning rather than demonstrating that the learned boundaries reflect semantic structure. Moreover, the information-theoretic formulation introduces a high-level semantic variable S, but its relationship with the learned latents is not directly evaluated. Additional analysis of the learned span boundaries, or comparisons with alternative semantic segmentation methods, would help substantiate the claim of semantic compression.

Questions for authors

See Weaknesses.

2 comments

Thanks to the response. Most of my concerns have been addressed. I will retain my current score.

Dear Reviewer hL13,

Thank you very much for your kind response and for taking the time to reconsider our rebuttal and revisions. We sincerely appreciate your acknowledgment that most of your concerns have been addressed.

Thank you once again for your valuable time and consideration!

Sincerely, The Authors of Submission #7368

Reviewer_PFRLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present CoLa2, a hierarchical language model that compresses variable-length text spans into continuous latent representations, generates these latents using a Diffusion Transformer (DiT) trained with Flow Matching, and converts them into text using a masked diffusion decoder.

Training proceeds in two stages. In the first stage, the encoder and decoder are jointly trained using reconstruction and generation-oriented objectives under different latent conditioning schemes. In the second stage, the encoder and decoder continue to be trained jointly, while the DiT fits the latent distribution with its gradients stopped at the encoder. During inference, each generated text block is re-encoded using the predicted span lengths, and these corrected latents are stored as the history for subsequent blocks.

The authors study architectural and training choices through ablations and theoretical analysis. They report scaling with training tokens on five downstream tasks and ablate module sizes.

Critical strengths
  1. The paper ablates segmentation, latent size, gradient routing, noise schedules and guidance. The uncompressed Cola-MDM baseline helps separate the effects of compression from decoder improvements.
  2. Figure 4 shows that joint encoder–decoder adaptation improves LAMBADA accuracy from 47.6% to 53.5%. Adding generation-oriented supervision raises it to 58.2% in the same setting.
  3. The authors report matching the scaling baselines in model size (about 2.4B parameters), training data and evaluation protocol.
Critical weaknesses
  1. My main concern is novelty. Cosine-based segmentation appears in H-Net [1] and a DLCM ablation [2]; joint latent-space training in LDLM [4]; continuous conditioning of MDMs in CRoCoDiL [5]; and latent robustness training in COSMOS [6]. The implementations differ, but the contribution seems mostly to combine these ideas within Cola DLM [7]. The rate controller also resembles loss-free MoE balancing [3], which isn’t cited.
  2. Table 2 tests detachment against gradient injection every 10 steps, not gradual warmup as in LDLM. The final gap is only 1 percentage point. Without results across seeds, I find this weak evidence for ruling out diffusion gradients. Table 3 also has gaps up to 1.4 points, which the authors treat as small.
  3. In Figure 10, Cola2-MDM finishes below AR and LLaDA [8] on average, while uncompressed Cola-MDM performs best. There are gains over AR on RACE and similar results on LAMBADA, but the overall case for compression needs measured efficiency gains. Appendix H gives analytical costs; latency, throughput and FLOP-matched comparisons are missing.
  4. The component ablations use LAMBADA only. The paper does not evaluate open-ended generation quality, likelihood, or contexts longer than the 512-token training limit.
  5. Appendix I.1 omits prompt templates, shot counts, detailed answer-extraction rules and evaluation splits, making the results hard to reproduce.
  6. Re-encoding is not ablated. Figure 9 tests latent corruption, which does not tell us whether storing re-encoded rather than proposed latents improves generation across successive blocks.

[1] Hwang et al. “Dynamic Chunking for End-to-End Hierarchical Sequence Modeling”. 2026

[2] Qu et al. “Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space”. 2025

[3] Wang et al. “Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts”. 2024

[4] Meshchaninov et al. “How to Train Your Latent Diffusion Language Model Jointly With the Latent Space”. 2026

[5] Uziel et al. “CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language”. 2026

[6] Meshchaninov et al. “Cosmos: Compressed and Smooth Latent Space for Text Diffusion Modeling”. 2025

[7] Guo et al. “Continuous Latent Diffusion Language Model”. 2026

[8] Nie et al. “Large Language Diffusion Models”. 2025

Questions for authors
  1. Could you compare full detachment with gradually increasing the DiT-to-encoder gradient, as in LDLM [4], while keeping the rest of the Cola2 setup fixed?
  2. Could you report standard deviations or confidence intervals across multiple random seeds for the main results and ablations, especially the small differences in Tables 2 and 3?
  3. Could you specify the prompting templates, few-shot counts, answer-extraction rules and evaluation splits?
  4. Could you report measured training costs, inference latency and throughput for Cola2-MDM, Cola-MDM, AR and LLaDA, including decoding, guidance and re-encoding costs? How do they compare at equal quality and at matched computational budgets?
  5. How much does re-encoding contribute? Please compare storing proposed latents with storing re-encoded latents, and measure generation quality and error accumulation as the number of generated blocks increases within the supported context length.
7 comments
AuthorsOpenReview ↗

Gentle Reminder: Follow-up on Our Rebuttal and Revised Manuscript

Dear Reviewer PFRL,

First and foremost, we would like to sincerely thank you once again for dedicating your valuable time to carefully reviewing our manuscript and providing such thorough, rigorous, and constructive feedback. We particularly appreciate your recognition of our systematic component ablations, the benefits of joint continuous–discrete learning, and our efforts to ensure fair experimental comparisons. Your thoughtful suggestions have been invaluable in helping us improve our work.

We have taken all your concerns very seriously, including the novelty of our approach, gradient training strategies and statistical reliability, the efficiency benefits of semantic compression, evaluation reproducibility, and the contribution of the re-encoding mechanism.

In our previous rebuttal, we have addressed these concerns point by point and made every effort to provide additional evidence, including new controlled experiments, multi-seed evaluations, measured inference efficiency, re-encoding ablations, and longer-context training results. We have also further clarified the distinctions between our method and related work. All corresponding additions and revisions have been incorporated into the latest revised manuscript, with changes highlighted in blue, and we would be sincerely grateful if you could kindly review them.

We sincerely hope that these additional efforts help clarify our contributions and adequately address your concerns. If you find that our responses, additional experiments, and manuscript revisions have sufficiently resolved your main concerns, we would be deeply grateful if you could kindly reconsider your assessment and score.

With fewer than three days remaining before the author–reviewer discussion deadline, we would greatly appreciate the opportunity to receive your further feedback. If any concerns remain unresolved, we would be very grateful for your guidance and would do our utmost to provide further clarification and improve the manuscript within the remaining discussion period.

Thank you once again for your valuable time, careful consideration, and constructive guidance.

Sincerely,
The Authors of Submission #7368

AuthorsOpenReview ↗

Final Gentle Reminder: Follow-up on Our Rebuttal — Submission #7368

Dear Reviewer PFRL,

First and foremost, we would like to sincerely thank you once again for your careful review of our manuscript and your valuable feedback. Your constructive suggestions have been extremely helpful in improving our work and strengthening our experimental evidence.

We have taken all your concerns very seriously and addressed them point by point in our previous rebuttal. We have also conducted substantial additional experiments, including multi-seed evaluations, comparisons of gradient training strategies, inference efficiency measurements, re-encoding ablations, and larger-scale pretraining results. All corresponding revisions have been incorporated into the latest manuscript and highlighted in blue.

With less than one day remaining before the author–reviewer discussion deadline, we would be very grateful for the opportunity to receive your further feedback. If you have any remaining concerns or questions that require clarification, we would be more than happy to provide additional explanations and continue the discussion.

We sincerely hope that our additional experiments and revisions have adequately addressed your main concerns. If you feel that these issues have been satisfactorily resolved, we would be deeply grateful if you could kindly reconsider your assessment and score.

Thank you once again for your valuable time and constructive feedback during this busy review period. We greatly appreciate the opportunity to engage in this discussion with you and sincerely look forward to hearing from you.

Sincerely, The Authors of Submission #7368

Reviewer_PFRLOpenReview ↗

Thank you for the detailed response and the additional experiments.

Thank you for the detailed response and the additional experiments. These additions address several points raised in my review, although I still have concerns about novelty given the overlap with prior work.

Regarding efficiency, you note that Table 7 does not establish end-to-end superiority. Could you clarify what one timed inference step includes and whether the measurements correspond to generating the same number of tokens?

Overall, the additional evidence is enough for me to raise my rating from 2 to 3.

Reviewer_uuFoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript introduce Cola2, a hierarchical architecture that combines continuous semantic compression with a MDM decoder. The authors usea an encoder to first compress the context into continuous latents, and then a MDM decoder uses these latents as conditioning to generate discrete text. During training, the authors first jointly train the encoder and MDM decoder, then introduce DiT training while continuing to update both the encoder and decoder. The authors also balance the decoder’s reconstruction ability and autonomous generation ability through mixed training under clean, noisy, and dropped latent conditions. The authors claim that the compress latent carries dense, predictive semantics, so the decoder can handle the residual uncertainty, thereby showing competitive performance and scaling potential on downstream tasks.

Critical strengths
  1. It is a meaningful attempt to use encoder to compress a latent from the context, and then let MDM use this as a condition to generate tokens.
  2. The paper addresses important training details through a joint learning mechanism for continuous and discrete representations. The overall framework is well developed, considers various aspects of training, and provides clear explanations and detailed descriptions.
Critical weaknesses

The benefits of the two-stage latent-to-text generation architecture are not yet clear. For example, in Figure 10, Cola2-MDM does not show a consistent overall advantage over the other methods. The experiment in Figure 9 show that the trained decoder uses the latents and is sensitive to their content alignment. However, they do not establish that adding a latent pathway is better than training a model without latent conditioning from scratch. The performance degradation may also reflect a dependence on this pathway developed during joint training.

Questions for authors
  1. Have you compared Cola2 with a MDM that uses the same block-causal attention structure but removes the latent conditioning under a matched total parameter count or training compute budget?

  2. Appendix H notes the generation compression rate is not guaranteed to match the training target. what latent-to-token ratio and span-length distribution are actually achieved during inference? Do they substantially across tasks or generation lengths?

1 comment
AuthorsOpenReview ↗

Gentle Reminder: Invitation for Further Feedback on Our Rebuttal

Dear Reviewer uuFo,

First and foremost, we would like to sincerely thank you once again for taking the time, despite your busy schedule, to carefully review our manuscript and provide such thoughtful and constructive feedback. We particularly appreciate your recognition of the value of compressing context into continuous latent representations to guide MDM generation, as well as our efforts in jointly learning continuous and discrete representations and systematically investigating key training mechanisms.

We greatly value the important concerns you raised, particularly regarding the practical advantages of the two-stage latent-to-text architecture over a pure MDM baseline, the quality–efficiency trade-off of semantic compression, and the actual compression ratios achieved during inference.

In our previous rebuttal, we have addressed these concerns point by point and made every effort to provide additional experiments and analyses, including a parameter-matched pure MDM baseline, measured inference latency, and actual compression ratios across different tasks. All these additional results and corresponding explanations have also been incorporated into the latest revised manuscript, with changes highlighted in blue, and we would sincerely appreciate it if you could kindly review them.

We sincerely hope that these additions help clarify both the contributions and limitations of our work and adequately address your concerns.

With fewer than three days remaining before the author–reviewer discussion deadline, we would be very grateful for the opportunity to receive your further feedback. If there are any remaining questions or aspects that require further clarification, we would sincerely appreciate your guidance and would be more than willing to make additional improvements within the remaining discussion period.

Thank you once again for your valuable time, encouraging assessment, and constructive suggestions.

Sincerely,
The Authors of Submission #7368

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.