- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes sentence curve language models (SCLMs), which introduce a fixed B-spline representation into diffusion language models. Instead of directly predicting token-position representations, the backbone predicts control points shared across nearby target positions. Experiments evaluate Gaussian diffusion models for English–German translation, diffusion language models on LM1B, and a preliminary block-diffusion extension.
Critical strengths
- A simple construction with evidence of applicability across diffusion formulations. The fixed input/output mappings introduce no trainable parameters and preserve the original objectives. The numerical-stability and hyperparameter analyses help identify practical limitations.
Critical weaknesses
-
The computational budget is not controlled in the comparison. SCLM typically uses 2–3 control points per token, increasing the sequence length processed by the backbone. Although Table 8 reports modest translation inference overhead, training costs and LM1B latency are not reported. A compute-matched baseline or a non-spline control with similarly expanded intermediate states is needed to distinguish the benefits of spline sharing from those of additional computation.
-
The language-modeling evaluation only measures sample quality. Gen. PPL is computed by an external GPT-2 Large evaluator; neither the model’s own held-out NLL/PPL nor cross-dataset zero-shot PPL is reported. The results therefore support improved generation quality, but do not establish better likelihood modeling or zero-shot generalization, which is the core metric for language modeling.
Questions for authors
- Likelihood evaluation: Can you report the model's cross-dataset zero-shot PPL following prior work (e.g. MDLM/BD3LM)? This would help establish whether the SCLM improves likelihood modeling, or if the improvements are limited to generation quality.
5 comments
Response to Reviewer A3Hg (2/3)
3. Generation at matched estimated FLOPs
Revised location: Appendix D.3, Table 11.
We added computation-matched comparisons by adjusting NFE in both directions. The estimate includes the dominant Transformer matrix operations, vocabulary projection, spline mappings, and final cleanup prediction. We define budgets using either model at NFE=64/128/256/512 and adjust the other model's NFE; all budget mismatches are below 0.83%.
The tables below report mean results, with standard deviations for MAUVE. TFLOPs are per generated sequence, and self-BLEU is expressed in percent. The anchor rows reuse the corresponding main-table results.
Budgets defined by SCLM
| Model | NFE | TFLOPs | Gen. PPL ↓ | Self-BLEU ↓ | MAUVE ↑ |
|---|---|---|---|---|---|
| SoftMask | 175 | 4.989 | 24.03 | 20.83 | .1325 ± .0075 |
| SCLM | 64 | 4.993 | 37.79 | 19.83 | .1354 ± .0156 |
| SoftMask | 349 | 9.922 | 18.69 | 22.73 | .1321 ± .0129 |
| SCLM | 128 | 9.909 | 24.52 | 22.10 | .1391 ± .0237 |
| SoftMask | 695 | 19.730 | 16.05 | 24.54 | .1187 ± .0126 |
| SCLM | 256 | 19.742 | 18.57 | 23.20 | .1427 ± .0377 |
| SoftMask | 1389 | 39.404 | 13.84 | 26.52 | .1085 ± .0086 |
| SCLM | 512 | 39.407 | 15.28 | 24.02 | .1360 ± .0320 |
SCLM achieves higher mean MAUVE at all four budgets. Increasing SoftMask's NFE from 175 to 1389 improves Gen. PPL from 24.03 to 13.84, but raises self-BLEU from 20.83% to 26.52% while MAUVE decreases from .1325 to .1085. This pattern is consistent with a quality–diversity trade-off limiting the benefit of additional denoising steps in the tested setting.
Budgets defined by SoftMask
| Model | NFE | TFLOPs | Gen. PPL ↓ | Self-BLEU ↓ | MAUVE ↑ |
|---|---|---|---|---|---|
| SoftMask | 64 | 1.843 | 40.48 | 18.90 | .1330 ± .0071 |
| SCLM | 23 | 1.844 | 115.48 | 15.17 | .1099 ± .0034 |
| SoftMask | 128 | 3.657 | 27.50 | 20.29 | .1340 ± .0066 |
| SCLM | 47 | 3.687 | 50.73 | 18.50 | .1353 ± .0088 |
| SoftMask | 256 | 7.285 | 20.82 | 21.96 | .1342 ± .0074 |
| SCLM | 94 | 7.298 | 29.11 | 21.33 | .1309 ± .0230 |
| SoftMask | 512 | 14.543 | 16.92 | 23.19 | .1257 ± .0126 |
| SCLM | 188 | 14.519 | 20.97 | 22.29 | .1253 ± .0200 |
These reverse comparisons characterize the trade-off introduced by reducing SCLM's denoising steps. Performance at a fixed FLOP budget depends on how computation is allocated.
Our proposed scaling direction is to invest additional computation in the sentence curve representation while retaining the number of denoising steps. At the four SCLM-defined budgets, this achieves higher mean MAUVE than allocating the same estimated FLOPs to additional SoftMask steps. The runtime measurements above show that the additional per-step computation entails only 4.10–4.77% extra inference latency at equal NFE on the RTX 4090. Together, these results support sentence curve prediction as a promising direction for scaling inference computation with modest measured runtime overhead under the tested conditions.
Response to Reviewer A3Hg (3/3)
Weakness 2 / Question 1: Held-out likelihood and cross-dataset evaluation
1. Model likelihood on LM1B and zero-shot OpenWebText
Additional results presented in this response; model and training settings are described in Appendix C.2.
We provide the requested likelihood evaluation using LM1B-trained MDLM, SoftMask, and SCLM checkpoints. We report recorded LM1B checkpoint-validation estimates and evaluate the checkpoints on a common held-out OpenWebText (OWT) corpus without fine-tuning, using the same tokenizer as the LM1B-trained models.
NLL is the model's native denoising-ELBO estimate per token, and estimated PPL is computed as . These measurements evaluate the models' own likelihood estimates, separately from Gen. PPL computed by the external GPT2-Large evaluator.
| Model | LM1B NLL ↓ | LM1B estimated PPL ↓ | OWT NLL ↓ | OWT estimated PPL ↓ |
|---|---|---|---|---|
| MDLM | 3.5397 | 34.46 | 4.8434 | 126.90 |
| SoftMask | 3.5275 | 34.04 | 4.8198 | 123.94 |
| SCLM | 3.5214 | 33.83 | 4.8438 | 126.95 |
The LM1B entries are recorded checkpoint-validation scores, including MDLM's training validation subset; the OWT entries use the common zero-shot evaluation above.
SCLM obtains the lowest recorded in-domain NLL and estimated PPL, complementing the generation-quality results. Its zero-shot OWT estimate is close to MDLM: 126.95 versus 126.90. Our central objective is to improve generation under the modeled data distribution, and these additional likelihood measurements broaden the assessment to both in-domain modeling and cross-domain transfer.
2. Generation on a second dataset
Revised location: Appendix D.9, Table 21.
We additionally trained and evaluated the models directly on OWT to test whether the generation benefits extend beyond LM1B. This experiment assesses generation on a second training distribution; the preceding experiment assesses zero-shot likelihood transfer from LM1B.
MDLM, SoftMask, and SCLM use the same tiny-DiT backbone with eight layers, hidden width 256, eight attention heads, and approximately 32.9M parameters. They share the training setup and are trained for 200K iterations, with EMA checkpoints selected by validation NLL. SCLM reuses its LM1B spline configuration: control-point ratio 3.0 and fixed degree 5.
Evaluation uses ten generation seeds and 1,024 samples per seed, with a common ReMDM-loop sampling setup. We report Gen. PPL from GPT2-Large, self-BLEU, and MAUVE, retaining MAUVE as the primary generation metric. The table gives mean results, with standard deviations for MAUVE; self-BLEU is in percent.
| NFE | Model | Gen. PPL ↓ | Self-BLEU ↓ | MAUVE ↑ |
|---|---|---|---|---|
| 64 | MDLM | 214.62 | 2.11 | .4610 ± .0536 |
| 64 | SoftMask | 185.12 | 2.26 | .5359 ± .0473 |
| 64 | SCLM | 185.98 | 2.23 | .6003 ± .0471 |
| 128 | MDLM | 174.31 | 2.28 | .5630 ± .0698 |
| 128 | SoftMask | 135.37 | 2.58 | .6793 ± .0519 |
| 128 | SCLM | 138.51 | 2.51 | .7217 ± .0578 |
| 256 | MDLM | 144.53 | 2.43 | .6368 ± .0410 |
| 256 | SoftMask | 103.97 | 2.87 | .7236 ± .0423 |
| 256 | SCLM | 107.45 | 2.74 | .7440 ± .0429 |
| 512 | MDLM | 118.73 | 2.68 | .6650 ± .0377 |
| 512 | SoftMask | 80.07 | 3.29 | .7687 ± .0346 |
| 512 | SCLM | 83.09 | 3.14 | .7826 ± .0422 |
SCLM achieves the highest mean MAUVE at every evaluated NFE on OWT, while improving Gen. PPL over MDLM throughout. Compared with SoftMask, it achieves lower self-BLEU at all four NFEs alongside higher mean MAUVE. These preliminary results extend the evidence for a favorable quality–diversity balance to web text dataset.
Response to Reviewer A3Hg (1/3)
Thank you for the thoughtful suggestions. We have revised the manuscript based on the additional experiments and clarifications below. We present the key results directly in this response and identify the corresponding revised sections. We also provide the requested held-out likelihood and zero-shot evaluation results here.
Weakness 1: Expanded states and computational cost
1. Benefits beyond state expansion
Revised location: Appendix D.4, Tables 12–13.
We added degree-1 expansion-only controls that retain the corresponding full SCLM's state count, backbone width, attention, positional treatment, and training/decoding settings. This directly tests whether simply processing more intermediate states explains the improvement.
On fully non-autoregressive LM1B, expansion-only and full SCLM both process 384 states for 128 tokens, use approximately 139M parameters, and retain the same SoftMask feedback mechanism. The expansion-only control uses degree 1, while full SCLM uses degree 5. In this LM1B control, different target positions have disjoint support in the fixed mapping, so the expanded backbone remains available without direct cross-target sharing through the mapping. Its self-attention can still learn dependencies.
The following table reports mean MAUVE over ten generation seeds, with 1,024 samples per seed. MAUVE is our primary generation metric for assessing the overall quality–diversity balance.
Full SCLM achieves higher mean MAUVE than the expansion-only control at every evaluated NFE. The identical state count and backbone width show that the expanded representation dimension alone does not account for these gains. The comparison supports the value of the spline configuration, which controls target sharing and the associated mapping geometry.
We also added expansion-only controls on IWSLT14, preserving the corresponding full SCLM's control-point ratio and backbone while removing or nearly eliminating direct cross-target sharing. Results are BLEU:
Without KD, state expansion itself contributes to the improvement, and full SCLM provides further gains of 0.65/0.82 BLEU in En→De/De→En. With KD, full SCLM improves over expansion-only by 0.84/0.32 BLEU. Thus, the additional benefits of the target-coupling configuration hold across both translation directions and both KD settings.
2. Measured inference and training cost
Revised location: Appendix D.3, Table 9 and the training-cost paragraph.
We added the requested LM1B inference and training benchmarks on an RTX 4090, using batch size 1 and sequence length 128. The inference table reports median seconds per generated sample after warm-up, including the final cleanup prediction and excluding model loading and text decoding.
Despite processing three times as many states, SCLM adds only 4.10–4.77% measured inference latency over SoftMask at equal NFE. For example, at NFE=256, mean MAUVE increases from .1342 to .1427 with 4.46% additional measured latency.
For training, both models use device/global batch size 1, no gradient accumulation, and the same BF16 mixed-precision setup. After 30 warm-up updates, 200 timed updates give:
The approximately 6.0% overhead covers the forward and backward passes, loss computation, gradient clipping, optimizer and scheduler steps, and EMA updates, with data preprocessing excluded.
These measurements show that the additional per-step arithmetic can be handled efficiently by the GPU implementation at this operating point. We next assess the generation quality obtained by allocating computation to sentence curve prediction.