- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DLCM forms variable-length latent segments from token representations, processes them with a wider Transformer, and generates tokens through causal cross-attention. It introduces global compression regularization, decoupled µP, and a compression-aware scaling law. After training on 1T tokens, the 2.3B model improves mean zero-shot accuracy over a 1.3B dense baseline by 1.01 percentage points, with estimated forward-FLOPs savings of 11.8% at 8K and 26.0% at 32K.
Critical strengths
1.Careful causal specification.
Concept availability is explicitly distinguished from segment membership, with consistent visibility rules during training and generation. The paper also acknowledges that replicated keys and values change softmax weighting rather than claiming equivalence to unique-concept attention.
2.Useful controlled findings.
At matched capacity, global regularization improves mean accuracy by 0.72 points and brings realized compression closer to its target. The learning-rate sweeps also support µP transfer across the evaluated widths.
Critical weaknesses
- Limited evidence for the architectural contribution.
The paper explicitly adopts H-Net’s boundary principle and ratio regularization, yet the main comparison is against a dense model rather than a hierarchical alternative. The results suggest a promising accuracy–estimated-compute trade-off, but do not establish the advantage of DLCM’s particular design over its closest predecessors.
- Scaling and allocation predictions lack independent validation.
The nine trajectories do not independently vary backbone allocation at each model size, leaving the separation of token- and concept-capacity effects insufficiently established. The reported fit checks are in-sample. Moreover, the allocation projections use a separate coefficient set whose fitting data are not clearly documented; the manuscript itself notes that trajectory-fit quality does not validate these larger-scale projections.
- Practical efficiency remains unverified.
The measured speedups concern an attention kernel, not end-to-end model throughput, latency, or memory. The headline 32K saving is a cost-model extrapolation beyond the 8K training context, without accompanying long-context quality evaluation.
Questions for authors
-
Can you provide a matched-compute comparison with H-Net or another hierarchical baseline, even at smaller scale, and isolate the contribution of the cross-attention decoder?
-
Which data produced the analytical-projection coefficients? Can held-out runs with independently varied model size and backbone allocation validate their predictions?
-
Can you report end-to-end throughput, latency, and memory, together with quality evaluation beyond 8K?
-
What are the held-out validation losses or perplexities of the two main models?
As the authors did not respond during the discussion period, my concerns remain unaddressed. I am therefore keeping my original score.