acceptodds
Under review as a conference paper at ICLR 2027

Sentence Curve Language Models

Abstract

Language models (LMs) are central to modern AI systems, and diffusion language models (DLMs) have recently emerged as a competitive alternative. Both use word embeddings not only as inputs but also, implicitly, as target representations. We argue that conventional targets primarily provide token-wise supervision and do not explicitly encourage coordinated prediction across neighboring positions, which may be particularly important under noisy parallel decoding. To address this, we propose sentence curve, a spline representation whose control points are shared across multiple target words, and introduce the sentence curve language model (SCLM), which extends DLMs to predict through this shared representation. Theoretically, we show that sentence curve prediction reweights token-wise gradients and characterize a trade-off between target-side coupling and representation capacity. Empirically, SCLM achieves state-of-the-art performance among DLMs on IWSLT14 and WMT14, surpasses autoregressive Transformers on WMT14, and trains effectively without knowledge distillation. On LM1B, SCLM further improves generic language generation in a standard-scale fully non-autoregressive setting and shows promising preliminary transfer to semi-autoregressive block diffusion.

Primary area:
generative models
Keywords:
Diffusion Language Models, Non-Autoregressive Generation, Target Representation, Structured Prediction, Sentence Representation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A3HgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes sentence curve language models (SCLMs), which introduce a fixed B-spline representation into diffusion language models. Instead of directly predicting token-position representations, the backbone predicts control points shared across nearby target positions. Experiments evaluate Gaussian diffusion models for English–German translation, diffusion language models on LM1B, and a preliminary block-diffusion extension.

Critical strengths
  • A simple construction with evidence of applicability across diffusion formulations. The fixed input/output mappings introduce no trainable parameters and preserve the original objectives. The numerical-stability and hyperparameter analyses help identify practical limitations.
Critical weaknesses
  • The computational budget is not controlled in the comparison. SCLM typically uses 2–3 control points per token, increasing the sequence length processed by the backbone. Although Table 8 reports modest translation inference overhead, training costs and LM1B latency are not reported. A compute-matched baseline or a non-spline control with similarly expanded intermediate states is needed to distinguish the benefits of spline sharing from those of additional computation.

  • The language-modeling evaluation only measures sample quality. Gen. PPL is computed by an external GPT-2 Large evaluator; neither the model’s own held-out NLL/PPL nor cross-dataset zero-shot PPL is reported. The results therefore support improved generation quality, but do not establish better likelihood modeling or zero-shot generalization, which is the core metric for language modeling.

Questions for authors
  • Likelihood evaluation: Can you report the model's cross-dataset zero-shot PPL following prior work (e.g. MDLM/BD3LM)? This would help establish whether the SCLM improves likelihood modeling, or if the improvements are limited to generation quality.
5 comments
AuthorsOpenReview ↗

Response to Reviewer A3Hg (1/3)

Thank you for the thoughtful suggestions. We have revised the manuscript based on the additional experiments and clarifications below. We present the key results directly in this response and identify the corresponding revised sections. We also provide the requested held-out likelihood and zero-shot evaluation results here.

Weakness 1: Expanded states and computational cost

1. Benefits beyond state expansion

Revised location: Appendix D.4, Tables 12–13.

We added degree-1 expansion-only controls that retain the corresponding full SCLM's state count, backbone width, attention, positional treatment, and training/decoding settings. This directly tests whether simply processing more intermediate states explains the improvement.

On fully non-autoregressive LM1B, expansion-only and full SCLM both process 384 states for 128 tokens, use approximately 139M parameters, and retain the same SoftMask feedback mechanism. The expansion-only control uses degree 1, while full SCLM uses degree 5. In this LM1B control, different target positions have disjoint support in the fixed mapping, so the expanded backbone remains available without direct cross-target sharing through the mapping. Its self-attention can still learn dependencies.

The following table reports mean MAUVE over ten generation seeds, with 1,024 samples per seed. MAUVE is our primary generation metric for assessing the overall quality–diversity balance.

ModelNFE=64NFE=128NFE=256NFE=512
SoftMask.1330.1340.1342.1257
Expansion-only SCLM.1335.1297.1244.1248
Full SCLM.1354.1391.1427.1360

Full SCLM achieves higher mean MAUVE than the expansion-only control at every evaluated NFE. The identical state count and backbone width show that the expanded representation dimension alone does not account for these gains. The comparison supports the value of the spline configuration, which controls target sharing and the associated mapping geometry.

We also added expansion-only controls on IWSLT14, preserving the corresponding full SCLM's control-point ratio and backbone while removing or nearly eliminating direct cross-target sharing. Results are BLEU:

ModelEn→De, KDDe→En, KDEn→De, without KDDe→En, without KD
Difformer27.5232.4222.9928.58
Expansion-only SCLM27.2632.2425.3930.25
Full SCLM28.1032.5626.0431.07

Without KD, state expansion itself contributes to the improvement, and full SCLM provides further gains of 0.65/0.82 BLEU in En→De/De→En. With KD, full SCLM improves over expansion-only by 0.84/0.32 BLEU. Thus, the additional benefits of the target-coupling configuration hold across both translation directions and both KD settings.

2. Measured inference and training cost

Revised location: Appendix D.3, Table 9 and the training-cost paragraph.

We added the requested LM1B inference and training benchmarks on an RTX 4090, using batch size 1 and sequence length 128. The inference table reports median seconds per generated sample after warm-up, including the final cleanup prediction and excluding model loading and text decoding.

ModelState lengthNFE=64NFE=128NFE=256NFE=512
MDLM1281.07802.12424.23368.5840
SoftMask1281.12852.22804.46468.9695
SCLM3841.18232.32594.66389.3370
SCLM overhead vs. SoftMask—+4.77%+4.39%+4.46%+4.10%

Despite processing three times as many states, SCLM adds only 4.10–4.77% measured inference latency over SoftMask at equal NFE. For example, at NFE=256, mean MAUVE increases from .1342 to .1427 with 4.46% additional measured latency.

For training, both models use device/global batch size 1, no gradient accumulation, and the same BF16 mixed-precision setup. After 30 warm-up updates, 200 timed updates give:

ModelTraining time, mean ± SD
SoftMask117.57 ± 8.02 ms/update
SCLM124.59 ± 10.41 ms/update

The approximately 6.0% overhead covers the forward and backward passes, loss computation, gradient clipping, optimizer and scheduler steps, and EMA updates, with data preprocessing excluded.

These measurements show that the additional per-step arithmetic can be handled efficiently by the GPU implementation at this operating point. We next assess the generation quality obtained by allocating computation to sentence curve prediction.

AuthorsOpenReview ↗

Response to Reviewer A3Hg (2/3)

3. Generation at matched estimated FLOPs

Revised location: Appendix D.3, Table 11.

We added computation-matched comparisons by adjusting NFE in both directions. The estimate includes the dominant Transformer matrix operations, vocabulary projection, spline mappings, and final cleanup prediction. We define budgets using either model at NFE=64/128/256/512 and adjust the other model's NFE; all budget mismatches are below 0.83%.

The tables below report mean results, with standard deviations for MAUVE. TFLOPs are per generated sequence, and self-BLEU is expressed in percent. The anchor rows reuse the corresponding main-table results.

Budgets defined by SCLM

ModelNFETFLOPsGen. PPL ↓Self-BLEU ↓MAUVE ↑
SoftMask1754.98924.0320.83.1325 ± .0075
SCLM644.99337.7919.83.1354 ± .0156
SoftMask3499.92218.6922.73.1321 ± .0129
SCLM1289.90924.5222.10.1391 ± .0237
SoftMask69519.73016.0524.54.1187 ± .0126
SCLM25619.74218.5723.20.1427 ± .0377
SoftMask138939.40413.8426.52.1085 ± .0086
SCLM51239.40715.2824.02.1360 ± .0320

SCLM achieves higher mean MAUVE at all four budgets. Increasing SoftMask's NFE from 175 to 1389 improves Gen. PPL from 24.03 to 13.84, but raises self-BLEU from 20.83% to 26.52% while MAUVE decreases from .1325 to .1085. This pattern is consistent with a quality–diversity trade-off limiting the benefit of additional denoising steps in the tested setting.

Budgets defined by SoftMask

ModelNFETFLOPsGen. PPL ↓Self-BLEU ↓MAUVE ↑
SoftMask641.84340.4818.90.1330 ± .0071
SCLM231.844115.4815.17.1099 ± .0034
SoftMask1283.65727.5020.29.1340 ± .0066
SCLM473.68750.7318.50.1353 ± .0088
SoftMask2567.28520.8221.96.1342 ± .0074
SCLM947.29829.1121.33.1309 ± .0230
SoftMask51214.54316.9223.19.1257 ± .0126
SCLM18814.51920.9722.29.1253 ± .0200

These reverse comparisons characterize the trade-off introduced by reducing SCLM's denoising steps. Performance at a fixed FLOP budget depends on how computation is allocated.

Our proposed scaling direction is to invest additional computation in the sentence curve representation while retaining the number of denoising steps. At the four SCLM-defined budgets, this achieves higher mean MAUVE than allocating the same estimated FLOPs to additional SoftMask steps. The runtime measurements above show that the additional per-step computation entails only 4.10–4.77% extra inference latency at equal NFE on the RTX 4090. Together, these results support sentence curve prediction as a promising direction for scaling inference computation with modest measured runtime overhead under the tested conditions.

AuthorsOpenReview ↗

Response to Reviewer A3Hg (3/3)

Weakness 2 / Question 1: Held-out likelihood and cross-dataset evaluation

1. Model likelihood on LM1B and zero-shot OpenWebText

Additional results presented in this response; model and training settings are described in Appendix C.2.

We provide the requested likelihood evaluation using LM1B-trained MDLM, SoftMask, and SCLM checkpoints. We report recorded LM1B checkpoint-validation estimates and evaluate the checkpoints on a common held-out OpenWebText (OWT) corpus without fine-tuning, using the same tokenizer as the LM1B-trained models.

NLL is the model's native denoising-ELBO estimate per token, and estimated PPL is computed as exp⁡(NLL)\exp(\mathrm{NLL}). These measurements evaluate the models' own likelihood estimates, separately from Gen. PPL computed by the external GPT2-Large evaluator.

ModelLM1B NLL ↓LM1B estimated PPL ↓OWT NLL ↓OWT estimated PPL ↓
MDLM3.539734.464.8434126.90
SoftMask3.527534.044.8198123.94
SCLM3.521433.834.8438126.95

The LM1B entries are recorded checkpoint-validation scores, including MDLM's training validation subset; the OWT entries use the common zero-shot evaluation above.

SCLM obtains the lowest recorded in-domain NLL and estimated PPL, complementing the generation-quality results. Its zero-shot OWT estimate is close to MDLM: 126.95 versus 126.90. Our central objective is to improve generation under the modeled data distribution, and these additional likelihood measurements broaden the assessment to both in-domain modeling and cross-domain transfer.

2. Generation on a second dataset

Revised location: Appendix D.9, Table 21.

We additionally trained and evaluated the models directly on OWT to test whether the generation benefits extend beyond LM1B. This experiment assesses generation on a second training distribution; the preceding experiment assesses zero-shot likelihood transfer from LM1B.

MDLM, SoftMask, and SCLM use the same tiny-DiT backbone with eight layers, hidden width 256, eight attention heads, and approximately 32.9M parameters. They share the training setup and are trained for 200K iterations, with EMA checkpoints selected by validation NLL. SCLM reuses its LM1B spline configuration: control-point ratio 3.0 and fixed degree 5.

Evaluation uses ten generation seeds and 1,024 samples per seed, with a common ReMDM-loop sampling setup. We report Gen. PPL from GPT2-Large, self-BLEU, and MAUVE, retaining MAUVE as the primary generation metric. The table gives mean results, with standard deviations for MAUVE; self-BLEU is in percent.

NFEModelGen. PPL ↓Self-BLEU ↓MAUVE ↑
64MDLM214.622.11.4610 ± .0536
64SoftMask185.122.26.5359 ± .0473
64SCLM185.982.23.6003 ± .0471
128MDLM174.312.28.5630 ± .0698
128SoftMask135.372.58.6793 ± .0519
128SCLM138.512.51.7217 ± .0578
256MDLM144.532.43.6368 ± .0410
256SoftMask103.972.87.7236 ± .0423
256SCLM107.452.74.7440 ± .0429
512MDLM118.732.68.6650 ± .0377
512SoftMask80.073.29.7687 ± .0346
512SCLM83.093.14.7826 ± .0422

SCLM achieves the highest mean MAUVE at every evaluated NFE on OWT, while improving Gen. PPL over MDLM throughout. Compared with SoftMask, it achieves lower self-BLEU at all four NFEs alongside higher mean MAUVE. These preliminary results extend the evidence for a favorable quality–diversity balance to web text dataset.

Reviewer_e1pxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new diffusion model where they introduce dependencies across token embeddings using a spline. Specifically, the authors take a training sequence, convert them to embedding sequence (dim: Ld), corrupt these, and then convert them to noisy control points (dim: Nd where N>L) using a predetermined matrix transformation which is inspired using Bspline curves. The predetermined matrix transformation is fixed for a given L, eta and N. This is this passed through transformer layers get the denoised control points. This is finally multiplied with the inverse matrix, to get the denoised embedding vectors and then train it using the standard diffusion ELBO loss.

Critical strengths

This is a model that attempts to introduce joint structure across token embeddings in order to maintain sentence level coherence. As is known, diffusion models built on per word/token embeddings lose (conditional) dependency across tokens when denoising/unmasking. This work suggests a plausible way to address this issue. I think this idea is potentially interesting.

Critical weaknesses

First, their transformer works in an expanded sequence length, because the L-length input sequence is converted into a N-length embedding sequence that is processed by the transformer (with N/L >= 2, per paper in section 5.1). This means that (a) the effective latent dimension (and thus, representation dimension) is increased compared to the base model, and (b) the attention computation has increased from L^2 to N^2. This brings up the question of whether the benefits are simply due to a higher capacity model compared to the MDLM baseline.

Second, the evaluations are limited. Beyond LM1B, it is worthwhile to show results with OWT as well as other recent baselines which try to model dependencies across tokens through other ways in latent space (e.g., Anchored Diffusion Language Model, NeurIPS 2025).

Questions for authors

I think it is important to show that the spline induced coupling across embeddings is responsible for improvements with respect to MDLM. To do that, the transformer effective latent dimension needs to be normalized. For instance, the MDLM should be (d’ x L) such that (d x N) = (d’ x L) (or some other ways to make sure that latent dimension is the same, e.g., set N_{ratio} = 1).

It is worthwhile to do an iso-compute analysis (i.e., keeping the same number to computations) with respect to the baseline (e.g., fewer denoising steps, or change latent dimension as above).

Also, I think that the evaluations are somewhat limited. Beyond LM1B, it is worthwhile to show results with OWT as well as other recent baselines as discussed above. Also, I would suggest showing entropy and Mauve for generation on LM1B and use the Remasking Diffusion Model (ReMDM) sampler (https://remdm.github.io in NeurIPS 2025) with MDLM.

7 comments
AuthorsOpenReview ↗

Response to Reviewer e1px (1/3)

Thank you for the thoughtful suggestions. We have revised the manuscript based on the additional experiments and clarifications presented below. We present the relevant results and explanations here so that this response can be read independently; the section references identify the corresponding manuscript revisions.

Weakness 1 / Question 1: Benefits beyond expanded representation dimension

Revised locations: Appendix D.4, Tables 12–13; Appendix D.10, Figure 7.

1. Controlling the expanded representation dimension

We address the proposed dimension control by comparing full SCLM with a degree-1 expansion-only SCLM. Both use the same number of internal states, backbone width, self-attention, positional treatment, and training/decoding settings. This keeps the representation dimension d×Nd\times N fixed while changing the spline configuration.

For fully non-autoregressive LM1B, both variants process 384 states for 128 tokens, with hidden width 768 and approximately 139M parameters, and retain the same SoftMask feedback. Expansion-only uses spline degree 1, whereas full SCLM uses degree 5. In this degree-1 control, different target positions have disjoint support in the fixed mapping: B⊤BB^\top B is diagonal and R(B)=1R(B)=1. The backbone still retains its ordinary self-attention, so it can learn dependencies while the fixed mapping provides no direct cross-target sharing.

The mean MAUVE results are:

ModelNFE=64NFE=128NFE=256NFE=512
SoftMask.1330.1340.1342.1257
Expansion-only SCLM.1335.1297.1244.1248
Full SCLM.1354.1391.1427.1360

Full SCLM improves mean MAUVE over the matched expansion-only control at every NFE. Because the expanded state count and backbone width are identical, these gains cannot be explained by representation dimension alone. They support the value of the spline configuration, which controls target sharing and the associated mapping geometry.

2. Translation controls and the fixed-ratio sweep

We also added expansion-only controls on IWSLT14, retaining the corresponding full SCLM's control-point ratio and backbone. The degree-1 mapping removes or nearly eliminates cross-target sharing. Results are SacreBLEU:

ModelEn→De, KDDe→En, KDEn→De, without KDDe→En, without KD
Difformer27.5232.4222.9928.58
Expansion-only SCLM27.2632.2425.3930.25
Full SCLM28.1032.5626.0431.07

Without KD, expansion itself contributes to the improvement, and full SCLM provides additional gains of 0.65/0.82 BLEU in En→De/De→En. With KD, full SCLM improves over expansion-only by 0.84/0.32 BLEU. Thus, the additional benefits hold across both translation directions and both KD settings.

The separate preliminary BD3-LM sweep in Figure 7 provides complementary evidence: at the same control-point ratio of 3, mean Gen. PPL improves from 208.22 at degree 1 to 201.69 at degree 9. This comparison keeps the control-point count fixed while changing the degree. Together, the controlled experiments support the contribution of the sentence curve configuration beyond simply expanding the processing space.

AuthorsOpenReview ↗

Response to Reviewer e1px (2/3)

Weakness 1 / Question 2: Runtime efficiency and matched computation

Revised location: Appendix D.3, Tables 9 and 11 and the training-cost paragraph.

1. Measured inference and training cost

We first measure wall-clock cost on the same RTX 4090, using batch size 1 and sequence length 128. The inference results below are median seconds per generated sample after warm-up. They include the final cleanup prediction and exclude model loading and text decoding.

ModelState lengthNFE=64NFE=128NFE=256NFE=512
MDLM1281.07802.12424.23368.5840
SoftMask1281.12852.22804.46468.9695
SCLM3841.18232.32594.66389.3370
SCLM overhead vs. SoftMask—+4.77%+4.39%+4.46%+4.10%

Despite processing three times as many states, SCLM adds only 4.10–4.77% measured inference latency over SoftMask at matched NFE. For example, at NFE=256, mean MAUVE increases from .1342 to .1427 with only 4.46% additional measured latency.

For training, we use device/global batch size 1, no gradient accumulation, and the same BF16 mixed-precision optimization setup. After 30 warm-up updates, 200 timed updates give:

ModelTraining time, mean ± SD
SoftMask117.57 ± 8.02 ms/update
SCLM124.59 ± 10.41 ms/update

The approximately 6.0% overhead includes the forward pass, loss computation, backward pass, gradient clipping, optimizer and scheduler steps, and EMA updates. These are complete single-example update timings with data preprocessing excluded.

The NFE labels count reverse updates, followed by one cleanup prediction; both the timings and the FLOP estimates include that final call. These measurements show that an efficient GPU implementation can accommodate SCLM's additional per-step computation with modest runtime overhead at this operating point. The FLOP-matched comparisons below examine the generation quality obtained by allocating computation in this way.

2. Matched estimated FLOPs

To complement the runtime measurements, we compare generation quality at approximately equal estimated computation by adjusting NFE in both directions. The estimate includes the dominant Transformer matrix operations, vocabulary projection, spline mappings, and final cleanup prediction. We define budgets using either model at NFE 64/128/256/512 and adjust the other model's NFE; all budget mismatches are below 0.83%.

The following tables report mean generation metrics. TFLOPs are per generated sequence, and self-BLEU is expressed in percent. The anchor rows reuse the corresponding main-table results.

Budgets defined by SCLM

ModelNFETFLOPsGen. PPL ↓Self-BLEU ↓MAUVE ↑
SoftMask1754.98924.0320.83.1325
SCLM644.99337.7919.83.1354
SoftMask3499.92218.6922.73.1321
SCLM1289.90924.5222.10.1391
SoftMask69519.73016.0524.54.1187
SCLM25619.74218.5723.20.1427
SoftMask138939.40413.8426.52.1085
SCLM51239.40715.2824.02.1360

SCLM achieves higher mean MAUVE in all four comparisons at these budgets. Increasing SoftMask's NFE from 175 to 1389 improves Gen. PPL from 24.03 to 13.84, but increases self-BLEU from 20.83% to 26.52% while MAUVE decreases from .1325 to .1085. This indicates a quality–diversity trade-off limiting the benefit of additional denoising steps in the tested setting.

Budgets defined by SoftMask

ModelNFETFLOPsGen. PPL ↓Self-BLEU ↓MAUVE ↑
SoftMask641.84340.4818.90.1330
SCLM231.844115.4815.17.1099
SoftMask1283.65727.5020.29.1340
SCLM473.68750.7318.50.1353
SoftMask2567.28520.8221.96.1342
SCLM947.29829.1121.33.1309
SoftMask51214.54316.9223.19.1257
SCLM18814.51920.9722.29.1253

These reverse comparisons characterize the trade-off when the FLOP budget is enforced by reducing SCLM's denoising steps. Our proposed scaling direction is to invest additional computation in the sentence curve representation while retaining the number of denoising steps. At the four SCLM-defined budgets, this achieves higher mean MAUVE than spending the same estimated FLOPs on additional SoftMask denoising steps. The runtime results above show that the added per-step computation entails only 4.10–4.77% extra inference latency at matched NFE on the RTX 4090. Together, these results support sentence curve prediction as a promising computational scaling direction whose additional arithmetic can be handled efficiently at runtime under the tested conditions.

AuthorsOpenReview ↗

Response to Reviewer e1px (3/3)

Weakness 2 / Question 3: ADLM, ReMDM, entropy, and OpenWebText

1. Additional baseline and ReMDM sampling

Revised locations: Main Table 2; Appendix C.2; Appendix D.8, Tables 17–18.

We added ADLM, preserving its anchor-and-denoiser architecture. Our implementation uses a 12-layer anchor and a 6-layer denoiser, both with hidden width 768 and 12 attention heads. The anchor's predictions condition the denoising network, and each generation step evaluates both networks. This gives approximately 232.5M parameters, compared with approximately 139M for MDLM, SoftMask, and SCLM, all at context length 128.

We use ReMDM-loop for fully non-autoregressive LM1B generation, with top- p=1.0p=1.0 and the common schedule

=(0.3,0.8,0.2,0.5).$$ Evaluation uses **ten generation seeds and 1,024 samples per seed and setting**. Gen. PPL is computed by GPT2-Large; MAUVE uses GPT2-Large features and the same fixed set of 1,024 LM1B validation references across models and seeds. We treat MAUVE as the primary metric for the overall quality–diversity balance. The updated LM1B MAUVE results are mean ± standard deviation: | Model | NFE=64 | NFE=128 | NFE=256 | NFE=512 | |---|---:|---:|---:|---:| | MDLM | .1097 ± .0096 | .1093 ± .0052 | .1018 ± .0026 | .1020 ± .0024 | | ADLM | .1278 ± .0085 | .1344 ± .0179 | .1225 ± .0257 | .1302 ± .0397 | | SoftMask | .1330 ± .0071 | .1340 ± .0066 | .1342 ± .0074 | .1257 ± .0126 | | SCLM | **.1354 ± .0156** | **.1391 ± .0237** | **.1427 ± .0377** | **.1360 ± .0320** | **SCLM achieves the highest mean MAUVE at every evaluated NFE**, including the refined NFE=512 result of .1360, compared with .1302 for ADLM, .1257 for SoftMask, and .1020 for MDLM. ## **2. Sample entropy** *Revised location: Appendix D.8, Table 17.* We added the requested entropy measurements. For each generated sequence of model-token IDs, we compute the empirical token frequencies $f_v$ and their Shannon entropy, then average over generated sequences. This measures within-sequence token diversity in nats. The following values are mean ± standard deviation across generation seeds: | Model | NFE=64 | NFE=128 | NFE=256 | NFE=512 | |---|---:|---:|---:|---:| | MDLM | 3.456 ± .025 | 3.144 ± .020 | 2.968 ± .029 | 2.876 ± .042 | | ADLM | 3.441 ± .021 | 3.208 ± .036 | 3.064 ± .028 | 3.011 ± .039 | | SoftMask | 3.633 ± .022 | 3.527 ± .019 | 3.435 ± .024 | 3.355 ± .020 | | SCLM | 3.448 ± .016 | 3.331 ± .034 | 3.259 ± .023 | 3.211 ± .028 | At NFE=512, SCLM retains higher sample entropy than MDLM and ADLM while achieving the highest mean MAUVE. We use entropy as a complementary diversity measure and MAUVE to assess the overall balance against reference text. ## **3. Generation on OpenWebText** *Revised location: Appendix D.9, Table 21.* We added a preliminary **OpenWebText (OWT)** study comparing MDLM, SoftMask, and SCLM under a common training setup. All three models use an eight-layer DiT with hidden width 256, eight attention heads, and approximately **32.9M parameters**, trained for **200K iterations**. They share the optimizer and training setup; EMA checkpoints are selected by validation NLL. SCLM reuses its LM1B spline configuration, **control-point ratio 3.0 and fixed degree 5**. As in the LM1B evaluation above, we use **ten generation seeds and 1,024 samples per seed**, with the same metric definitions. The table reports mean results, with standard deviations included for MAUVE; self-BLEU is in percent and entropy in nats. | NFE | Model | Gen. PPL ↓ | Self-BLEU ↓ | MAUVE ↑ | Entropy | |---:|---|---:|---:|---:|---:| | 64 | MDLM | 214.62 | 2.11 | .4610 ± .0536 | 4.314 | | 64 | SoftMask | 185.12 | 2.26 | .5359 ± .0473 | 4.293 | | 64 | SCLM | 185.98 | 2.23 | **.6003 ± .0471** | 4.300 | | 128 | MDLM | 174.31 | 2.28 | .5630 ± .0698 | 4.292 | | 128 | SoftMask | 135.37 | 2.58 | .6793 ± .0519 | 4.255 | | 128 | SCLM | 138.51 | 2.51 | **.7217 ± .0578** | 4.263 | | 256 | MDLM | 144.53 | 2.43 | .6368 ± .0410 | 4.265 | | 256 | SoftMask | 103.97 | 2.87 | .7236 ± .0423 | 4.214 | | 256 | SCLM | 107.45 | 2.74 | **.7440 ± .0429** | 4.224 | | 512 | MDLM | 118.73 | 2.68 | .6650 ± .0377 | 4.235 | | 512 | SoftMask | 80.07 | 3.29 | .7687 ± .0346 | 4.169 | | 512 | SCLM | 83.09 | 3.14 | **.7826 ± .0422** | 4.172 | **SCLM achieves the highest mean MAUVE at every evaluated NFE on OWT**, while improving Gen. PPL over MDLM throughout. Compared with SoftMask, it achieves lower self-BLEU and higher token entropy across all four NFEs, with slightly higher Gen. PPL. These results extend the generation evidence to web text dataset.
Reviewer_5ubxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Sentence Curve Language Models (SCLMs), which represent a sequence of target embeddings through B-spline control points. Noised embeddings are mapped into control-point space using a pseudoinverse; the existing denoising backbone processes these states, and a fixed spline basis maps its predictions back to token positions for the original objective. The paper interprets this representation as an inductive bias for coordinated target prediction and analyzes its gradient geometry and correction cost. Experiments report improvements over Difformer on IWSLT14 and WMT14, improvements in mean generation-quality metrics on LM1B, and preliminary transfer to block diffusion.

Critical strengths
  1. A simple representation change produces meaningful translation improvements. The fixed input/output mappings introduce no additional trainable parameters and retain the underlying DLM objective. In Table 1, SCLM improves over the re-evaluated Difformer by 1.21/1.37 SacreBLEU on WMT14 En→De/De→En, and by 3.05/2.49 on IWSLT14 without knowledge distillation. These are the strongest empirical results in the paper and suggest that the intervention is practically useful, particularly when the target distribution has not been simplified by distillation. The additional baseline rescaling-factor tuning in Appendix C.2 strengthens the within-study comparison.
  2. The paper provides a useful analytical description and tests more than one diffusion setting. The gradient identity and minimum-norm correction formula in Section 4.3 and Appendix A.3 explain concrete consequences of inserting the spline map. The implementation is also evaluated in Gaussian, masked, and block-diffusion settings, and the appendices examine numerical stability and hyperparameter sensitivity. Although these results do not establish the proposed mechanism by themselves, the combination makes the idea more informative than an isolated architectural modification.
Critical weaknesses
  1. The contribution of spline-induced cross-token sharing is not sufficiently separated from that of processing more internal states. SCLM adds no trainable parameters, but its backbone processes two to three times as many states in the selected translation settings; the fully non-AR LM1B configuration processes 384 control-point states for 128 tokens. Thus, some gains could arise from the expanded intermediate representation and additional computation, rather than specifically from different token positions sharing spline control points. This is an alternative explanation to test, not an established explanation of the results. Appendix D.8/Figure 7 does provide a useful joint sweep of control-point count and spline degree in the preliminary BD3-LM setting. However, changing the count also changes the basis geometry, and the manuscript does not explicitly identify which low-degree configurations, if any, have disjoint support across token positions and therefore serve as a matched expansion-only control. Such a control could retain the backbone's ordinary self-attention while removing cross-token sharing in the fixed mapping. The selected IWSLT14 De→En configuration also has R̄(B)=1.0000 in Table 9, making clarification of nearly uncoupled configurations especially relevant. Finally, Appendix D.4 measures dCor after applying B; shared linear mixtures can themselves produce dependent responses. Higher dCor therefore does not by itself establish that learned coordination, rather than state expansion or the mapping operation, explains the quality gains.
  2. The robustness and computational cost of the LM1B gains are not yet adequately characterized. The MAUVE means favor SCLM at every reported NFE, which is positive evidence. At NFE=256, the increase from 0.1092 to 0.1308 is approximately 19.8% relative to SoftMask and should not be dismissed as negligible. However, Table 11 reports standard deviations of 0.0228 and 0.0468 over only three generation seeds, with 1,024 samples per seed; these are not confidence intervals or evidence of stability across independent training runs. The between-model difference needs its own uncertainty assessment rather than an inference from overlap of the marginal standard deviations. The main-table BD3-LM evaluations use 256 generated sequences and one generation seed per setting, limiting evidence about repeatability; Appendix D.8 separately provides a three-seed configuration sweep, so this limitation does not apply to all BD3-LM experiments. In addition, matching parameters, updates, and NFEs does not match computation when SCLM processes more internal states. Table 8 reports translation latency, but no analogous LM1B cost comparison is provided. The relationship between the discrete SCLM row and SoftMask feedback, the spline construction, and the MAUVE evaluation settings also need clarification. These gaps limit conclusions about robustness and cost-adjusted benefit without negating the observed mean improvements.
Questions for authors
  1. How much of the improvement requires cross-token sharing, beyond expansion of the intermediate states? Please first clarify whether any low-degree configurations in Figure 7 have disjoint support across token positions, using their basis overlap or R(B), and what those existing comparisons establish. If the existing experiments do not isolate this effect, a comparison on one existing translation setting with the same state count, backbone attention, positional treatment, and training/decoding settings, but no cross-token sharing in the fixed mapping, would be informative. Please compare translation quality and dCor and explain the selected De→En configuration with R̄(B)=1.0000.
  2. How robust is the LM1B improvement over SoftMask? Please distinguish training seeds from generation seeds, and provide uncertainty for the between-model MAUVE difference, using paired repetitions where appropriate. An analysis of existing outputs would already be useful; the reported marginal standard deviations do not determine this difference's uncertainty.
  3. What exactly is the discrete SCLM configuration and its cost? Please clarify whether the SCLM row uses SoftMask feedback, specify the B-spline knot/boundary construction and positional treatment of control points, and provide the MAUVE feature model, reference sampling, and generation settings. Please also report training cost and decoding latency for the compared LM1B checkpoints under the same hardware and batch setting.
  4. How should the theoretical assumptions be interpreted for the actual trained models? In Proposition 1/Appendix A.3, is the conditional-centering assumption an exact property of the finite vocabulary or an expectation over random embeddings? If the latter, the exact finite weighted cancellation needs justification. Please also distinguish the proven nested-enrichment result from the empirical behavior obtained by changing the number of spline control points.
8 comments
AuthorsOpenReview ↗

Response to Reviewer 5ubx (1/6)

Thank you for the thoughtful questions. We have revised the manuscript based on the additional experiments, analyses, and implementation and theoretical clarifications presented below. We present the relevant results and explanations here so that this response can be read independently; the section references identify the corresponding manuscript revisions.

Weakness 1 / Question 1: Target coupling beyond state expansion

1. Matched expansion-only controls

Revised locations: Appendix D.4, Tables 12–13; Appendix D.10, Figure 7.

We added degree-1 expansion-only SCLMs that retain the corresponding full SCLM's state count, backbone width, self-attention, positional treatment, and training/decoding settings. The intervention changes the spline configuration while keeping the expanded processing space.

For a nonnegative spline basis, disjoint support across target positions makes the Gram matrix B⊤BB^\top B diagonal and gives R(B)=1R(B)=1. The degree-1 De→En control has R(B)=1R(B)=1; the En→De control has Rˉ(B)=1.0000007\bar R(B)=1.0000007, indicating negligible aggregate overlap. Thus, these controls remove or nearly eliminate cross-target sharing in the fixed mapping while preserving the backbone's ordinary self-attention.

The IWSLT14 results are:

ModelEn→De, KDDe→En, KDEn→De, without KDDe→En, without KD
Difformer27.5232.4222.9928.58
Expansion-only SCLM27.2632.2425.3930.25
Full SCLM28.1032.5626.0431.07

Without KD, expansion alone contributes to the improvement, and full SCLM provides further gains of 0.65/0.82 BLEU in En→De/De→En. With KD, full SCLM improves over expansion-only by 0.84/0.32 BLEU. The additional gains therefore hold in all four settings.

For fully non-autoregressive LM1B generation, both SCLM variants process 384 control-point states for 128 target tokens, use the same approximately 139M-parameter backbone, and retain SoftMask feedback. Expansion-only uses degree 1, whereas full SCLM uses degree 5. Their mean MAUVE results are:

ModelNFE=64NFE=128NFE=256NFE=512
SoftMask.1330.1340.1342.1257
Expansion-only SCLM.1335.1297.1244.1248
Full SCLM.1354.1391.1427.1360

Full SCLM achieves higher mean MAUVE than the matched expansion-only control at every NFE. Together with the translation results, this supports the value of the spline configuration beyond processing more states.

Figure 7 provides complementary evidence at a fixed control-point ratio of 3: degree 1 has disjoint target supports and gives mean Gen. PPL 208.22, compared with 201.69 at degree 9. The broader sweep also shows why coupling and capacity must be considered together: at a control-point ratio of 1, increasing degree from 1 to 3 changes Gen. PPL from 212.80 to 226.53. Increasing coupling is therefore not uniformly beneficial; its effectiveness depends on the representation available to accommodate token-specific corrections.

AuthorsOpenReview ↗

Response to Reviewer 5ubx (2/6)

2. What R(B)R(B) and dCor measure

Revised locations: Section 4.3, after Lemma 1; Appendix D.5, Table 14.

Our hypothesis is that appropriate target-side dependency can improve generation. dCor describes coordinated prediction behavior; maximizing it does not guarantee better performance.

For a target-space gradient GG, the control-point gradient is GB⊤GB^\top. Lemma 1 defines

\quad R(B)\leq\kappa(B)^2, \quad \kappa(B)=\frac{s_{\max}(B)}{s_{\min}(B)}.$$ $R(B)$ compares squared gradient magnitudes for a normalized sequence-wide pattern and an average token-local pattern. It characterizes the fixed mapping, while dCor measures trained-model behavior. **$R(B)=1$ does not prevent learned dependencies** through self-attention and training data. For dCor, we fix the decoder diffusion input and perturb encoder states with Gaussian noise normalized by their average norm, or perturb source tokens. We measure Difformer's final hidden states and $\hat P_\theta B$ for SCLM, placing all models in the same target-position space. We average pairwise dCor between distinct positions, compute normalized AUCs, and average the two perturbation types. The task-selected KD configurations give: | Direction | Model | $\bar R(B)$ | $\bar\kappa(B)$ | dCor AUC | BLEU | |---|---|---:|---:|---:|---:| | En→De | Difformer | 1.0000 | 1.0000 | .6214 | 27.52 | | En→De | Expansion-only | 1.0000 | 1.1175 | .6450 | 27.26 | | En→De | Full SCLM | 1.3549 | 1.5698 | .6627 | **28.10** | | De→En | Difformer | 1.0000 | 1.0000 | .5633 | 32.42 | | De→En | Expansion-only | 1.0000 | 1.3035 | .6313 | 32.24 | | De→En | Full SCLM | 1.0000 | 1.1042 | .6089 | **32.56** | $\bar R(B)$ and $\bar\kappa(B)$ are respectively arithmetic and geometric means over training target lengths. Both variants have higher dCor than Difformer. Relative to expansion-only, full SCLM **increases dCor in En→De and decreases it in De→En while improving BLEU in both**. This is consistent with dependency modulation, including potentially moderating excessive coordination from expansion. The useful level may depend on remaining target variation and multimodality; this is an interpretation, not a causal conclusion from dCor. Expansion-only can learn substantial dependency through shared supervision of control points reconstructing the same target embedding and self-attention across positions. Its $\kappa(B)>1$ indicates unequal spectral scaling: disjoint columns can have different norms. It does not establish a preference for global coordination. For selected De→En, $\bar R(B)=1.0000064942$, rounded to 1.0000. Its degree is $\max(2,\lfloor0.01N\rfloor)$; almost all excess $\bar R(B)-1$ arises from $L\geq100$. Length averaging therefore conceals selective coupling on longer targets. Across 14 full-SCLM configurations, standardizing within each direction and pooling gives Pearson correlations with dCor of **0.688/0.751** for $\bar R(B)/\bar\kappa(B)$, and Spearman correlations of 0.459/0.534. These associations support using spline geometry to modulate dependency. ## **3. Does the deterministic mapping itself inflate dCor?** *Revised location: Appendix D.5, “Shuffled mapping-only analysis,” Table 15.* Shared control points could introduce dependence through $B$ itself. We disrupt joint variation while preserving the mapping. For each sentence and perturbation level, we collect 32 predicted control-point sequences and independently permute the perturbation indices for each control point. This preserves its vectors and empirical distribution. We then reapply the unchanged $B$, allowing dependence from shared mapping to remain. For Difformer, we independently shuffle embedding vectors at each target position. We use 10 validation batches, diffusion step 1000, and 30 shuffle repetitions. Original AUCs are recomputed alongside shuffles; full-SCLM rows average seven configurations per direction. | Direction | Model | Original AUC | Shuffled AUC | Difference | Shuffled / original | |---|---|---:|---:|---:|---:| | En→De | Difformer | .6218 | .4674 | .1544 | .7517 | | En→De | Expansion-only | .6450 | .4661 | .1789 | .7226 | | En→De | Full SCLM | .6796 | .4839 | .1957 | .7120 | | De→En | Difformer | .5651 | .4242 | .1409 | .7506 | | De→En | Expansion-only | .6313 | .4621 | .1692 | .7319 | | De→En | Full SCLM | .6221 | .4610 | .1611 | .7411 | Difformer retains approximately **75%** of its original AUC without a spline mapping. Both SCLM variants retain similar or smaller proportions (**71.2–74.1%**), with no consistent increase from expansion-only to full SCLM. This suggests no disproportionate inflation relative to the baseline in this analysis, although the ratios do not isolate $B$'s contribution.
AuthorsOpenReview ↗

Response to Reviewer 5ubx (3/6)

Weakness 2 / Question 2: Uncertainty and repeatability

Revised locations: Main Table 2; Appendix D.8, Tables 17–20.

1. Fully non-autoregressive LM1B

We expanded evaluation to ten generation seeds per trained checkpoint, with 1,024 samples per seed and setting. These are generation repetitions for fixed checkpoints, and characterize generation variability separately from variability across training runs.

We directly assess the uncertainty of the between-model MAUVE difference. For Δ=mˉSCLM−mˉSoftMask\Delta=\bar m_{\mathrm{SCLM}}-\bar m_{\mathrm{SoftMask}}, we use two-sided unpaired Welch confidence intervals, with standard error

\sqrt{\frac{s_{\mathrm{SCLM}}^2}{n_{\mathrm{SCLM}}} +\frac{s_{\mathrm{SoftMask}}^2}{n_{\mathrm{SoftMask}}}},$$ and the Welch–Satterthwaite degrees of freedom. Thus, the comparison uses uncertainty in the difference itself rather than overlap between the models' marginal standard deviations. | NFE | SoftMask MAUVE, mean ± SD | SCLM MAUVE, mean ± SD | $\Delta$ | 90% CI for $\Delta$ | 95% CI for $\Delta$ | |---:|---:|---:|---:|---:|---:| | 64 | .1330 ± .0071 | **.1354 ± .0156** | +.002355 | [−.007249, .011959] | [−.009371, .014081] | | 128 | .1340 ± .0066 | **.1391 ± .0237** | +.005061 | [−.008974, .019097] | [−.012171, .022294] | | 256 | .1342 ± .0074 | **.1427 ± .0377** | +.008464 | [−.013611, .030539] | [−.018703, .035630] | | 512 | .1257 ± .0126 | **.1360 ± .0320** | +.010328 | [−.009115, .029771] | [−.013456, .034112] | Differences and intervals use the unrounded results. **The mean advantage favors SCLM at every NFE**, including the refined NFE=512 evaluation. The intervals include zero, so we present these as consistent mean improvements with quantified generation uncertainty. ## **2. Preliminary semi-autoregressive BD3-LM** We also expanded the main-table BD3-LM evaluation to **three generation seeds and 1,024 samples per seed**. The preliminary tiny-DiT models are trained for 100K updates. The mean ± SD and Welch intervals for $\Delta=\mathrm{SCLM}-\mathrm{BD3\text{-}LM}$ are: | Block size | Context length | BD3-LM MAUVE | SCLM MAUVE | $\Delta$ | 90% CI | 95% CI | |---:|---:|---:|---:|---:|---:|---:| | 4 | 128 | .7592 ± .0457 | **.8572 ± .0112** | +.0980 | [.0242, .1718] | [−.0077, .2037] | | 8 | 128 | .7582 ± .0527 | **.8201 ± .0235** | +.0619 | [−.0193, .1431] | [−.0494, .1732] | | 16 | 128 | .7239 ± .0159 | **.7408 ± .0283** | +.0169 | [−.0264, .0602] | [−.0412, .0750] | | 4 | 256 | .8226 ± .0208 | **.8707 ± .0344** | +.0481 | [−.0046, .1008] | [−.0222, .1184] | | 4 | 512 | .8010 ± .0394 | **.9160 ± .0380** | +.1150 | [.0476, .1824] | [.0272, .2028] | **Mean MAUVE improves in all five configurations.** The largest gain is +0.1150 at block size 4/context length 512, whose 95% interval is entirely above zero. At block size 4/context length 128, the 90% interval is also entirely positive. These results strengthen the evidence for repeatability of the preliminary block-diffusion improvements.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary (2/2)

2. Computational comparisons and practical efficiency

  • Raised issue: e1px requested equal-computation comparisons. 5ubx requested comparable training and inference timings, while A3Hg highlighted missing cost measurements and requested computational controls. (Reviewers: 5ubx, e1px, A3Hg)
  • How we addressed it: We added runtime measurements and comparisons at matched estimated computation by adjusting generation steps. Our method achieves higher average quality–diversity scores when the baseline receives additional steps to match SCLM’s computational budget. In these comparisons, the baseline’s additional steps improve its likelihood-based generation score but reduce sample diversity. Conversely, reducing our method’s steps to match the baseline’s budget generally favors the baseline on the quality–diversity measure. Runtime measurements show modest overhead in the tested batch-size-one GPU setting.
  • Discussion history: 5ubx explicitly acknowledged that the runtime measurements addressed the corresponding questions. The reviewer also questioned whether the mixed equal-computation results sufficiently establish greater computational effectiveness. We acknowledged the dependence on budget allocation and clarified the specific scaling benefit supported by the results. No further response from e1px to these experiments appears in the discussion record. A3Hg’s overall acknowledgment stated that most concerns were addressed, without specifying individual items.
  • Our interpretation: These results support the proposed method as a useful way to allocate additional computation in the evaluated settings. Although they do not establish superiority at every budget, increasing the baseline’s generation steps did not match our method’s average quality–diversity scores at the tested SCLM-defined budgets. The runtime measurements additionally show that this allocation can be accommodated with modest latency overhead under the tested conditions. The remaining disagreement concerns whether this evidence sufficiently establishes the method’s computational benefit.

3. Evaluation coverage and uncertainty

  • Raised issue: 5ubx requested uncertainty estimates and clarification of generation versus training repetitions. e1px requested additional baselines, another dataset, diversity measurements, and an alternative sampling procedure. A3Hg requested model-likelihood and cross-dataset evaluation. (Reviewers: 5ubx, e1px, A3Hg)
  • How we addressed it: We expanded repeated generation evaluations, added confidence intervals, incorporated recent baselines and the requested sampling procedure, and conducted experiments on another text benchmark. These experiments extend the evidence for higher average generation quality while making uncertainty explicit, including confidence intervals containing zero for the mean score differences in the fully parallel comparison with the main baseline. We also provided model-likelihood evaluations to distinguish generation improvements from cross-dataset generalization.
  • Discussion history: A3Hg stated that the clarifications and additional experiments addressed most concerns. This acknowledgment concerned the response as a whole and did not identify which individual concerns were resolved. No further response from e1px to the additional baselines, dataset, sampling procedure, and diversity measurements appears in the discussion record.

4. Theoretical and analytical clarification

  • Raised issue: 5ubx questioned the interpretation of the dependency measurements, the mathematical assumptions, the scope of the theoretical results, and implementation details. (Reviewer: 5ubx)
  • How we addressed it: We added diagnostic comparisons and clarified the distinction between effects of the fixed representation and behavior learned during training. We also clarified the mathematical assumptions, the forms of representation expansion covered by the proofs, and the implementation. The analysis treats dependency as a descriptive property, without assuming that stronger dependency always improves performance.
  • Discussion history: 5ubx explicitly acknowledged that the clarifications of the mathematical assumptions and the scope of the expansion result addressed those questions. A3Hg also offered an overall acknowledgment that most concerns were addressed, without specifically commenting on these theoretical issues.
AuthorsOpenReview ↗

Rebuttal Summary (1/2)

We thank the reviewers for their thoughtful feedback. Below, we first summarize the strengths they recognized, then discuss the main concerns, the additional evidence we provided, and the subsequent exchanges, distinguishing the reviewers’ acknowledgments from our interpretation.

Recognized Strengths

The reviewers recognized several strengths of the proposed approach. e1px described the idea as potentially interesting and a plausible way to introduce dependencies between token representations for sentence-level coherence. 5ubx and A3Hg highlighted the simplicity of the construction: the fixed representation mappings introduce no additional trainable parameters and preserve the underlying models’ training objectives. Both reviewers also noted the evidence of applicability across different diffusion-model formulations.

5ubx emphasized the meaningful translation improvements, particularly without knowledge distillation, and noted that the additional baseline tuning strengthens the comparisons. The reviewer also recognized the value of the mathematical analysis in explaining concrete consequences of the proposed representation, and explicitly acknowledged the approach’s novelty during the discussion. A3Hg highlighted the numerical-stability and hyperparameter analyses as useful for understanding the method’s practical limitations. These assessments recognize contributions in the representation design, analytical understanding, and empirical applicability.

Discussions on Main Concerns

1. Contribution beyond simply processing more internal states

  • Raised issue: All reviewers asked whether the improvements come from the proposed coupling between token representations or simply from processing more internal states and using additional computation. (Reviewers: 5ubx, e1px, A3Hg)
  • Suggested checks: 5ubx proposed preserving the expanded processing space while removing the proposed coupling. A3Hg suggested either a similarly expanded control or an equal-computation comparison. e1px proposed increasing the representation dimension of a standard baseline or evaluating our method without state expansion.
  • How we addressed it: Following the control suggested by 5ubx and the corresponding alternative offered by A3Hg, we added expansion-only models that preserve the processing architecture and experimental settings while removing direct coupling through the fixed mapping. Expansion alone improves performance in some settings, while full SCLM provides additional machine translation gains and higher average quality–diversity scores in language generation.
  • Discussion history: 5ubx explicitly acknowledged that the proposed method performs better than simply expanding the internal states, following the expansion-only comparison they originally suggested. In the subsequent discussion, 5ubx raised an additional concern about comparisons against standard models of comparable capacity, referring to e1px’s suggestion. e1px acknowledged understanding the experimental setup, but questioned the strength of the expansion-only control and the fairness of comparisons with standard baselines. We explained why we consider the expansion-only versus full-SCLM comparison directly relevant to the original attribution question. We also clarified that the preliminary block-diffusion experiments include evaluation without state expansion, addressing the suggested setting. No further reply from e1px to our final clarification appears in the discussion record. A3Hg stated that the clarifications and additional experiments addressed most concerns, without specifying individual items.
  • Our interpretation: The expansion-only comparison provides direct evidence for the additional contribution of the coupling-inducing representation under the same processing architecture. A larger standard baseline would test a complementary scaling strategy, whereas the expansion-only control more directly examines the representation change within SCLM’s expanded architecture. In our view, the existing comparison addresses the original attribution question; the remaining disagreement concerns whether an additional architectural comparison is necessary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.