acceptodds
Under review as a conference paper at ICLR 2027

CARD: Dense Denoising for Causal Language Models

Abstract

Autoregressive language models use causal computation efficiently but generate one token at a time. Masked diffusion permits parallel generation, although through a different training interface. Applying denoising to a causal model creates an uneven-context problem: the same corruption can remove most of the useful history for one target and little for another. Causal Autoregressive Diffusion (CARD) addresses this problem while retaining the standard next-token interface. It predicts every clean token from a corrupted left prefix, using Soft Tail Masking to preserve evidence within the corrupted suffix and context-aware weights to account for the amount and location of missing history. In matched 1B-parameter experiments, CARD exceeds block diffusion (BD3LM) by 5.10 percentage points in eight-task accuracy and trains about three times as fast. Ablations attribute the gain to both mechanisms, and prediction-level analysis finds less disagreement across corrupted prefixes. The same checkpoint supports KV-cached sequential generation and confidence-based parallel decoding, with parallelism selected at inference time.

Primary area:
generative models
Keywords:
Large Language Models, Text Diffusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2phNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CARD, a training framework that enables denoising-based parallel generation while retaining standard causal attention and the next-token prediction interface. CARD combines dense supervision over all clean tokens, Soft Tail Masking to preserve local evidence within a corrupted suffix, and context-aware loss weighting that accounts for the quantity, recency, and consecutiveness of missing preceding tokens.

Critical strengths
  1. The paper provides well-controlled comparisons with matched training settings and extensive ablations. The results support the effectiveness of both Soft Tail Masking and context-aware weighting.

  2. CARD enables sequential and parallel decoding from the same checkpoint without architectural changes. The theoretical analysis of prediction consistency is also interesting.

Critical weaknesses
  1. The practical advantage over standard AR remains unclear. CARD achieves lower downstream accuracy and slower training than AR, despite improving clean-prefix perplexity. The reported decoding speedups are measured against sequential decoding of CARD itself, rather than a separately trained AR model with optimized decoding. Consequently, it is difficult to assess whether CARD offers a practical advantage over existing causal generation methods.
  2. The connection between the proposed training components and parallel decoding performance is insufficiently established. Although the ablations support their effectiveness on downstream accuracy, their impact on the speed‒quality trade-off of parallel decoding is not evaluated. Furthermore, Soft Tail Masking produces a different training distribution from the consecutive all-mask blocks used at inference. While Appendix C.3 quantifies the occurrence of such mask patterns, the effect of this mismatch on decoding quality, especially for larger blocks, remains unclear.
Questions for authors
  1. Can you compare CARD with a separately trained standard AR model using optimized decoding at comparable generation quality? How does CARD compare with related causal parallel-decoding approaches in terms of the speed‒quality trade-off?
  2. How do Soft Tail Masking and context-aware weighting individually affect parallel decoding performance? Can the authors evaluate the same parallel decoder on the existing ablation checkpoints to demonstrate their contributions?
  3. How does the mismatch between Soft Tail training masks and all-mask inference blocks affect decoding performance, particularly for larger block sizes? Can its effect be distinguished from other factors such as forced commitments and iteration caps?
1 comment

Thank you for these concrete suggestions. They helped us turn the original single-checkpoint decoding demonstration into a more informative speed-quality comparison. The revised Section 5.1 and Appendix F contain new experiments for each of the three issues you raised.

Comparison with optimized AR. We trained an independent AR baseline and evaluated verified speculative decoding under the same batch-128 throughput measurement. It obtains 24.54 TriviaQA F1 at 5,675 tokens/s. CARD-16 gives 6,629 tokens/s at 21.10 F1, and CARD-32 gives 14,769 tokens/s at 18.56 F1; CARD-8 gives 5,418 tokens/s at 21.51 F1. This identifies a real throughput advantage at larger blocks, together with its QA cost. At near-AR QA quality, the optimized AR baseline remains stronger. We also report matched Eso-LM and SBD adaptations and the native MDLM/BD3LM decoding results rather than relying only on a CARD-versus-CARD speedup (Appendix F.4).

Do the two training components help the decoder? The direct ablation now answers this. With identical eight-token decoding, uniform denoising obtains 19.91 TriviaQA F1, adding context weights reaches 20.18, Soft Tail with unit weights reaches 21.01, and full CARD reaches 21.51. Strict-tail masking reaches 20.02. The corresponding HellaSwag Qwen continuation scores are 11.25, 10.99, 10.90, 10.47, and 10.80, respectively (lower is better). Throughput is similar across these causal checkpoints, supporting an improvement in parallel prediction quality rather than a different decoding budget. Appendix F.3 includes 16- and 32-token results, WikiText-103, Rep-4, and output entropy.

What is the effect of the mask mismatch? We trained additional block-aligned corruption arms and compared them with Soft Tail on a 400-example success diagnostic, holding both block size and iteration cap fixed within each pair. For a 32-token block and eight iterations, success increases from 50.83% to 59.50% under block-aligned training, while forced commitments change from 22.58% to 21.52%. At 16 iterations, success is 59.17% versus 63.08%. We also vary the iteration cap independently, so these differences cannot be attributed to extra refinement passes. The complete factorial appears in Appendix F.5.

The results establish both effects: Soft Tail improves the trained model's parallel predictions, and aligning the training corruption more closely with an all-mask block can improve larger-block recovery further. We hope these direct controls make the contribution and the remaining quality-speed trade-off easier to evaluate, and address the issues behind your soundness assessment.

Reviewer_66BmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Causal Autoregressive Diffusion (CARD), a pretraining objective that trains a standard causal Transformer (with an ordinary lower-triangular attention mask and one-token input shift) to predict every clean token $ x_n $ from a corrupted prefix in which a subset of tokens has been replaced by [MASK]. The authors argue that naive causal denoising suffers from an uneven-context problem—early tokens have short histories easily wiped out by uniform masking, while later tokens lose all local context under contiguous suffix masking—and introduce Soft Tail Masking and Context-Aware Reweighting techniques.

Critical strengths
  1. Zero-overhead architectural compatibility with standard causal LMs. The method changes nothing about the Transformer architecture, attention kernel, KV cache layout, or next-token head; it is purely a training-time input-corruption and loss-weighting scheme that preserves all standard AR infrastructure. Unlike BD3LM (which doubles training sequence length to 2L2L with block-sparse attention) or WeDLM (which requires dual-stream training), a single unmodified causal checkpoint can be used for standard sequential decoding or parallel block decoding—which would be a strong practical advantage if parallel decoding were convincingly demonstrated.
  2. Experimental control Ten objectives ×\times three paired seeds ×\times 300B tokens at 1B parameters—with shared initialization, identical token stream ordering, five intermediate checkpoints, per-seed task vectors, and 110M factorial controls—is well above the empirical standard of most pretraining papers.
Critical weaknesses
  1. Novelty is narrower than presented, and the most relevant prior work is not discussed. Training a causal next-token language model on randomly masked inputs with zero architectural overhead is discussed in Forgetful Causal Masking (FCM; Liu et al., 2022, arXiv:2210.13432) and Mask-Enhanced Autoregressive Prediction (MEAP; Zhuang et al., ICML 2025, arXiv:2502.07490). Furthermore, causal-attention masked diffusion with exact KV caching and a single checkpoint supporting both sequential and parallel generation is Eso-LM (Sahoo et al., 2025, arXiv:2506.01928) and Set Block Decoding (SBD; Gat et al., 2025, arXiv:2509.04185). None of these works are cited or compared, leaving the paper's incremental contribution restricted to two heuristics (Soft Tail Masking and Context-Aware Reweighting) on top of FCM/MEAP.

  2. Unvalidated link between the proposed training components and parallel decoding, alongside missing diffusion parallel-decoding comparisons. The paper motivates Soft Tail Masking and Context-Aware Reweighting as enabling parallel decoding, yet evaluates all training ablations (Tables 1–3) only on clean-prefix AR metrics. Because both mechanisms push the objective closer to clean-prefix AR, it seems that the main accuracy gain stems from CARD's AR nature rather than diffusion. Meanwhile, Section 5 (i) never compares parallel decoding against BD3LM, MDLM, or AR + Lookahead, (ii) includes no task needing bidirectional context (infilling, constrained generation, planning), and (iii) no diversity or sampling-mode evaluation (only Rep-4 and agreement with greedy sequential). The practical case is not established, which together with Weakness 1 leads me to recommend rejection.

  3. Presentation requires substantial revision. While the individual equations are readable and the appendix tables are extensive, the Introduction and Methodology require substantial revision in relation to relevant works and how the method is framed and explained, and several key experimental specifications are unclear or missing.

Questions for authors

Please address the points raised in Critical Weaknesses above, as well as the following specific clarification questions:

  1. Positioning relative to prior masked-causal and hybrid AR-diffusion work: Please position CARD relative to FCM (Liu et al., 2022, arXiv:2210.13432) and MEAP (Zhuang et al., 2025, arXiv:2502.07490), Eso-LM (Sahoo et al., 2025, arXiv:2506.01928) and Set Block Decoding (Gat et al., 2025, arXiv:2509.04185) , and Diffusion Forcing (Chen et al., NeurIPS 2024). How does CARD compare empirically to Eso-LM and SBD on the speed–quality frontier?
  2. Parallel decoding on ablated checkpoints and diffusion baselines: What are the CARD-8, CARD-16, and CARD-32 generation metrics (TriviaQA F1, HellaSwag/WikiText-103, Rep-4, and generation entropy) when applied to the matched 1B checkpoints for (a) Strict tail (λ=1\lambda=1), (b) Soft Tail, unit weights, and (c) Uniform denoising, and how do they compare against native parallel decoding on the matched 1B BD3LM and MDLM checkpoints?
  3. BD3LM HellaSwag anomaly and MDLM step-time overhead:
    • (a) Why does BD3LM score 43.22% on HellaSwag at 300B tokens—6.24 pp below MDLM (49.46%)—while BD3LM outperforms MDLM in Arriola et al. (2025)?
    • (b) Why does MDLM take 3.28 s/step vs. 1.90 s/step for AR on the identical 33-layer, L=2048L=2048 model? Did MDLM use the same FlashAttention-2 kernel as used in AR? and why is CARD 15% slower than AR with identical attention?
1 comment

Thank you for the detailed review. Your concerns about the closest prior work and the missing parallel-decoding ablations were important, and they led to substantial changes in the manuscript. In particular, we agree that the original positioning did not give FCM, MEAP, Eso-LM, and Set Block Decoding sufficient attention. The Introduction and Related Work now discuss these methods alongside Diffusion Forcing. We distinguish the established ideas of masked causal prediction and cached parallel generation from CARD's specific contribution: fixed-budget Soft Tail corruption, weighting by the missing causal context of each target, and measured gains from combining them. The new Section 5.1 and Appendix F test that distinction directly.

Matched prior-work comparisons. We trained a separate three-seed, 1B-parameter suite for 300B tokens per run, including matched adaptations of FCM, MEAP, Eso-LM, and SBD. On the common eight-task evaluation, the four adaptations reach 51.36%, 52.38%, 52.40%, and 52.40%, compared with 53.38% for CARD (Appendix F.1). This places the proposed objective against the relevant causal approaches under a controlled budget. The adaptations are not evaluations of those works' released checkpoints, and the revised table labels them accordingly.

Do the components improve parallel generation? Yes. We ran the same confidence decoder at block sizes 8, 16, and 32 on uniform denoising, uniform plus weighting, Soft Tail with unit weights, strict-tail masking, and full CARD checkpoints. At eight tokens their TriviaQA F1 scores are 19.91, 20.18, 21.01, 20.02, and 21.51, respectively. CARD also has the best HellaSwag Qwen continuation score among these five objectives (10.47, lower is better), while throughput remains nearly unchanged. Both ingredients therefore improve the predictions used by parallel decoding, beyond their gains on clean-prefix accuracy (Appendix F.3).

We also evaluated the native parallel decoders of the retrained MDLM, BD3LM, Eso-LM, and SBD arms. At eight tokens, their TriviaQA F1 / throughput pairs are 19.59 / 4,289, 19.59 / 6,290, 21.92 / 5,876, and 21.77 / 5,574, compared with CARD's 21.51 / 5,418. CARD gives a better HellaSwag Qwen continuation score (10.47 versus 12.72, 13.05, 10.82, and 10.81). Eso-LM and SBD offer stronger eight-token QA efficiency, while CARD offers better quality under this external continuation metric. Appendix F.4 includes larger blocks, so the comparison does not depend on a single decoding configuration.

Speed and quality against optimized AR. We added a separately trained AR checkpoint with verified speculative decoding, which reaches 24.54 TriviaQA F1 at 5,675 tokens/s. CARD-16 reaches 6,629 tokens/s at 21.10 F1, and CARD-32 reaches 14,769 tokens/s at 18.56 F1. CARD-8 runs at 5,418 tokens/s and 21.51 F1. These results show where parallel denoising becomes faster and how much QA quality it costs. We have stated the boundary plainly: optimized AR remains preferable when its higher QA accuracy is required; CARD provides faster lower-F1 operating points with a standard causal checkpoint.

Other generation settings. To address the question about tasks beyond greedy continuation, Appendix F.7 adds constrained JSON generation, short arithmetic-planning prompts, prompted infilling, and nucleus-sampling diversity. We describe the infilling setup precisely: the right context is supplied in the prompt, so it is not native bidirectional infilling. For CARD-8, nucleus sampling raises Distinct-2 from 23.64% to 52.24%, lowers self-BLEU-4 from 1.000 to 0.618, and lowers Rep-4 from 2.59% to 1.56%. Rep-4 and output entropy for the component ablations and native decoders are also reported.

HellaSwag and step times. We checked whether the BD3LM/MDLM gap came from the answer-scoring convention. On a common 1,000-question subset, length-normalized HellaSwag accuracy remains 44.87% for BD3LM and 49.87% for MDLM; we also report raw-sum scoring (Appendix F.6). The original ordering is therefore not an artifact of using raw rather than length-normalized candidate scores. Our controlled BD3LM implementation uses its specified 32-token block and noise schedule, and we now identify this protocol when comparing with Arriola et al.

Appendix F.6 also breaks down 500 post-warmup updates per seed. Total step times are 1,912 ms for AR, 2,207 ms for CARD, 3,289 ms for MDLM, and 6,475 ms for BD3LM. AR and CARD have causal attention; MDLM computes bidirectional denoising attention; BD3LM uses structured block attention. The timings are consequently not an equal-kernel benchmark. Of CARD's 295 ms difference from AR, 167 ms falls in the attention bucket, 36 ms in the input/mask bucket, and 92 ms in remaining phases. This accounts for the measured end-to-end gap without treating the attention buckets as isolated kernel benchmarks.

We hope this new evidence addresses the two main reasons behind your soundness concern.

Reviewer_5mphOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CARD trains a standard causal Transformer to predict every clean token from a corrupted left context. It combines Soft Tail masking with context-aware loss weights and retains the shifted next-token interface. Experiments train 1B models on 300B FineWeb tokens with three paired seeds and evaluate eight downstream tasks, component controls, and decoding strategies. CARD improves average task accuracy over BD3LM while remaining below AR. The same checkpoint supports cached sequential generation and confidence-based parallel decoding with a speed–quality tradeoff.

Critical strengths
  1. The training contribution is supported by three paired seeds, a 2×2 component study, and positional-weight controls. Table 2 reports average accuracies of 50.20, 51.75, 51.90, and 53.13 for uniform/unit, uniform/weights, Soft Tail/unit, and CARD, respectively. These controls support the usefulness of both components.

  2. CARD retains the standard causal architecture and requires no additional prediction head. The evaluation reports both efficiency gains and quality costs: CARD-8 achieves 1.41× throughput while TriviaQA F1 changes from 21.94 to 21.64. The paper also acknowledges AR’s accuracy and training-speed advantages, helping readers assess the method’s scope.

Critical weaknesses

Table 6 establishes parallel decoding on a CARD checkpoint, but does not isolate the contribution of Soft Tail and context-aware weighting to that capability. The selection protocol for the conservative decoding settings also needs clarification.

Questions for authors

Were the confidence threshold and iteration caps in Appendix E.1 fixed in advance, calibrated on separate data, or selected using the reported generation-evaluation outputs?

1 comment

Thank you for the positive assessment of our paired-seed study and for identifying the two questions left open by Table 6. We have addressed both in the revised Section 5.1 and Appendix F, with a separate decoding study on the retrained 1B checkpoints.

Decoder selection. The original Table 6 used a common confidence threshold of 0.90, with iteration caps of 8, 16, and 8 for block sizes 8, 16, and 32. We have now made the distinction between that same-checkpoint illustration and the calibrated follow-up explicit. For the new comparison, we ran a separate 600-question validation sweep over thresholds 0.78, 0.84, 0.88, 0.90, 0.94, and 0.97. We chose the fastest setting that met the stated F1-drop budget relative to sequential decoding, giving 0.84 for 8- and 16-token blocks and 0.78 for 32-token blocks. These choices were made from validation results, not the reported follow-up evaluation outputs. The caps, thresholds, and selection objective are now specified in Appendix F.2.

Contribution to parallel decoding. Under the identical eight-token decoder, TriviaQA F1 rises from 19.91 for uniform denoising to 20.18 with context weights, 21.01 with Soft Tail and unit weights, and 21.51 with full CARD. The corresponding HellaSwag Qwen continuation scores improve from 11.25 to 10.99, 10.90, and 10.47 (lower is better), with throughput near 5.4k tokens/s throughout. This connects the component gains to parallel generation itself, which the original clean-prefix ablations could not establish. Appendix F.3 also gives the strict-tail control and the 16- and 32-token results.

We hope the new calibration details and decoder ablations resolve the remaining concerns you raised about the original generation study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.