- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes CARD, a training framework that enables denoising-based parallel generation while retaining standard causal attention and the next-token prediction interface. CARD combines dense supervision over all clean tokens, Soft Tail Masking to preserve local evidence within a corrupted suffix, and context-aware loss weighting that accounts for the quantity, recency, and consecutiveness of missing preceding tokens.
Critical strengths
-
The paper provides well-controlled comparisons with matched training settings and extensive ablations. The results support the effectiveness of both Soft Tail Masking and context-aware weighting.
-
CARD enables sequential and parallel decoding from the same checkpoint without architectural changes. The theoretical analysis of prediction consistency is also interesting.
Critical weaknesses
- The practical advantage over standard AR remains unclear. CARD achieves lower downstream accuracy and slower training than AR, despite improving clean-prefix perplexity. The reported decoding speedups are measured against sequential decoding of CARD itself, rather than a separately trained AR model with optimized decoding. Consequently, it is difficult to assess whether CARD offers a practical advantage over existing causal generation methods.
- The connection between the proposed training components and parallel decoding performance is insufficiently established. Although the ablations support their effectiveness on downstream accuracy, their impact on the speed‒quality trade-off of parallel decoding is not evaluated. Furthermore, Soft Tail Masking produces a different training distribution from the consecutive all-mask blocks used at inference. While Appendix C.3 quantifies the occurrence of such mask patterns, the effect of this mismatch on decoding quality, especially for larger blocks, remains unclear.
Questions for authors
- Can you compare CARD with a separately trained standard AR model using optimized decoding at comparable generation quality? How does CARD compare with related causal parallel-decoding approaches in terms of the speed‒quality trade-off?
- How do Soft Tail Masking and context-aware weighting individually affect parallel decoding performance? Can the authors evaluate the same parallel decoder on the existing ablation checkpoints to demonstrate their contributions?
- How does the mismatch between Soft Tail training masks and all-mask inference blocks affect decoding performance, particularly for larger block sizes? Can its effect be distinguished from other factors such as forced commitments and iteration caps?
Thank you for these concrete suggestions. They helped us turn the original single-checkpoint decoding demonstration into a more informative speed-quality comparison. The revised Section 5.1 and Appendix F contain new experiments for each of the three issues you raised.
Comparison with optimized AR. We trained an independent AR baseline and evaluated verified speculative decoding under the same batch-128 throughput measurement. It obtains 24.54 TriviaQA F1 at 5,675 tokens/s. CARD-16 gives 6,629 tokens/s at 21.10 F1, and CARD-32 gives 14,769 tokens/s at 18.56 F1; CARD-8 gives 5,418 tokens/s at 21.51 F1. This identifies a real throughput advantage at larger blocks, together with its QA cost. At near-AR QA quality, the optimized AR baseline remains stronger. We also report matched Eso-LM and SBD adaptations and the native MDLM/BD3LM decoding results rather than relying only on a CARD-versus-CARD speedup (Appendix F.4).
Do the two training components help the decoder? The direct ablation now answers this. With identical eight-token decoding, uniform denoising obtains 19.91 TriviaQA F1, adding context weights reaches 20.18, Soft Tail with unit weights reaches 21.01, and full CARD reaches 21.51. Strict-tail masking reaches 20.02. The corresponding HellaSwag Qwen continuation scores are 11.25, 10.99, 10.90, 10.47, and 10.80, respectively (lower is better). Throughput is similar across these causal checkpoints, supporting an improvement in parallel prediction quality rather than a different decoding budget. Appendix F.3 includes 16- and 32-token results, WikiText-103, Rep-4, and output entropy.
What is the effect of the mask mismatch? We trained additional block-aligned corruption arms and compared them with Soft Tail on a 400-example success diagnostic, holding both block size and iteration cap fixed within each pair. For a 32-token block and eight iterations, success increases from 50.83% to 59.50% under block-aligned training, while forced commitments change from 22.58% to 21.52%. At 16 iterations, success is 59.17% versus 63.08%. We also vary the iteration cap independently, so these differences cannot be attributed to extra refinement passes. The complete factorial appears in Appendix F.5.
The results establish both effects: Soft Tail improves the trained model's parallel predictions, and aligning the training corruption more closely with an all-mask block can improve larger-block recovery further. We hope these direct controls make the contribution and the remaining quality-speed trade-off easier to evaluate, and address the issues behind your soundness assessment.