acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Large Concept Models: Latent Reasoning in an Adaptive Semantic Space

Abstract

Large language models apply their full computational stack to every token, although the information carried by a sequence varies across spans. We introduce Dynamic Large Concept Models (DLCM), which form variable-length concepts from contextual token representations and process them with a larger concept-level backbone. A lightweight causal decoder uses completed concepts to generate subsequent tokens autoregressively, sharing concept-level computation across multiple predictions. Global compression regularization allows segment lengths to adapt to content while regulating their average. We also introduce a compression-aware scaling law that separates token capacity, concept capacity, and data, together with a decoupled maximal-update parametrization for modules of different widths. A 2.3B-parameter DLCM improves the mean accuracy over a 1.3B dense baseline by 1.01 percentage points across twelve zero-shot benchmarks, with approximately 34% lower estimated inference FLOPs. Both models are trained on one trillion tokens. Gains are strongest on commonsense and question-answering tasks, while compression and token-loss analyses reveal sensitivity to segment granularity. The scaling analysis further predicts lower loss at equal compute through concept-level capacity allocation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Concept Models,Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zNEfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DLCM forms variable-length latent segments from token representations, processes them with a wider Transformer, and generates tokens through causal cross-attention. It introduces global compression regularization, decoupled µP, and a compression-aware scaling law. After training on 1T tokens, the 2.3B model improves mean zero-shot accuracy over a 1.3B dense baseline by 1.01 percentage points, with estimated forward-FLOPs savings of 11.8% at 8K and 26.0% at 32K.

Critical strengths

1.Careful causal specification.

Concept availability is explicitly distinguished from segment membership, with consistent visibility rules during training and generation. The paper also acknowledges that replicated keys and values change softmax weighting rather than claiming equivalence to unique-concept attention.

2.Useful controlled findings.

At matched capacity, global regularization improves mean accuracy by 0.72 points and brings realized compression closer to its target. The learning-rate sweeps also support µP transfer across the evaluated widths.

Critical weaknesses
  1. Limited evidence for the architectural contribution.

The paper explicitly adopts H-Net’s boundary principle and ratio regularization, yet the main comparison is against a dense model rather than a hierarchical alternative. The results suggest a promising accuracy–estimated-compute trade-off, but do not establish the advantage of DLCM’s particular design over its closest predecessors.

  1. Scaling and allocation predictions lack independent validation.

The nine trajectories do not independently vary backbone allocation at each model size, leaving the separation of token- and concept-capacity effects insufficiently established. The reported fit checks are in-sample. Moreover, the allocation projections use a separate coefficient set whose fitting data are not clearly documented; the manuscript itself notes that trajectory-fit quality does not validate these larger-scale projections.

  1. Practical efficiency remains unverified.

The measured speedups concern an attention kernel, not end-to-end model throughput, latency, or memory. The headline 32K saving is a cost-model extrapolation beyond the 8K training context, without accompanying long-context quality evaluation.

Questions for authors
  1. Can you provide a matched-compute comparison with H-Net or another hierarchical baseline, even at smaller scale, and isolate the contribution of the cross-attention decoder?

  2. Which data produced the analytical-projection coefficients? Can held-out runs with independently varied model size and backbone allocation validate their predictions?

  3. Can you report end-to-end throughput, latency, and memory, together with quality evaluation beyond 8K?

  4. What are the held-out validation losses or perplexities of the two main models?

1 comment

As the authors did not respond during the discussion period, my concerns remain unaddressed. I am therefore keeping my original score.

Reviewer_hMcPOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Large Concept Models (DLCM), which is a hierarchical language model. It groups variable-length tokens into latent concepts and feeds these compressed representations into a larger Transformer. The authors also propose global compression regularization, a compression-aware scaling law, and decoupled μP optimization to better balance computation and model capacity. Experimental results show that the proposed architecture slightly surpasses a dense baseline by 1% with lower FLOPs.

Critical strengths
  1. The proposed hierarchical structure lets a bigger Transformer work on compressed sequences, which is a practical way to trade model capacity for compute cost.

  2. The paper studies scaling behavior, capacity allocation, optimization and compute overhead, giving helpful observations for hierarchical language modeling.

Critical weaknesses
  1. The core motivation and architecture is too similar to earlier work like H-Net and BLT. While the authors introduce several modifications, the changes here seem to be incremental, and the paper does not compare directly against these prior methods. For example, tt lacks comparisons against fixed or random segmentation to show that dynamic concept learning is necessary.

  2. The efficiency claim relies on FLOPs estimates instead of real end-to-end runtime. I expect the authors to supplement some further end-to-end runtime comparison with counterparts.

  3. The scaling law predictions have not been tested on other unseen model sizes.

Questions for authors

See above.

1 comment

As the authors did not engage in a discussion and did not respond during the discussion period, my concerns remain unaddressed. I am therefore keeping my original score.

Reviewer_Kvr9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Dynamic Large Concept Models, which pool tokens using their contextual representations into bigger units referred to as "concepts". These concepts are the input to a decoder model that autoregressively generates the next tokens. The paper shows FLOP efficiency and improvements on commonsense and question-answering tasks, and a scaling analysis that further predicts lower loss at equal compute. However, compression and token-loss analyses reveal sensitivity to segment granularity.

Critical strengths
  1. Finding more meaningful units for text is an interesting question. In the era of "scale is all you need", it is refreshing to see works that try to understand the data (i.e., text) better and use that to improve models, also at a few-billion-parameter size.

  2. The authors handle causality carefully, as the model only uses concepts once they are available. Moreover, the authors openly explain a limitation of their attention mechanism.

  3. Training on 1 trillion tokens is substantial. With stronger controls, this could be a solid contribution to systems and optimization.

Critical weaknesses
  1. The main gains are not convincingly attributable to the dynamic concepts introduced in this paper. The model has substantially more parameters than the baseline, and there is no matched-compute comparison or any ablation of key architectural changes. Fixed-chunk baselines (e.g., using POS) and even Oracle LLM-based chunking would be useful.

  2. The scaling-law claims are poorly supported. The model is fitted with more parameters than there are independent training trajectories, making its predictions unreliable. The claimed equal-compute advantage appears to come primarily from a lower fitted loss floor, rather than demonstrated improvements in scaling behavior.

Questions for authors
  1. Which models and evaluation protocol underlie Table 3 vs Table 2?

  2. How were the analytical projection coefficients in Table 7 obtained?

  3. Would the authors consider adding parameter-matched and FLOPs-matched dense baselines?

  4. Would the authors consider adding fixed-chunk and Oracle chunking baselines?

1 comment

The authors did not engage in a discussion, so my concerns were not addressed and thus I am keeping my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.