- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies self-conditioning (SC) and embedding geometry in CDLMs. It introduces DoubleCat, a two-level corruption process aimed at reducing training–inference mismatch, and SemiCat, which combines DoubleCat with clean-token reveals. It also investigates how training objectives shape embedding geometry and explores cheaper alternatives to SC in Plaid.
Critical strengths
I appreciate the paper goal to better understand the SC role in CDLMs and find simpler and more natural alternatives to this trick.
I view DoubleCat is a simple and elegant modification motivated by analysing categorical predictions during inference. It largely improves one-hot CDLMs and outperforms the baseline w/ SC without requiring an extra training forward pass. I consider it a valuable contribution to this setting, even though SC provides further improvements.
SemiCat demonstrates significant further empirical gains of combining DoubleCat with the revealing mechanism from CANDI. The analytical posterior used as a cheap conditioning signal is also an interesting exploratory contribution.
Critical weaknesses
W1 | I find several central takeaways in Sections 3 and 4 closely revisit prior work, making it difficult to identify what new understanding the analysis adds:
Section 3.1: Shabalin et al. (2026, §3.3) already connect sampling failures to low-density regions rarely visited during training and interpret SC as helping mitigate this problem.
Section 3.2: CANDI already identifies token-identifiability problems under Gaussian corruption and introduces clean-token reveals to provide reliable conditioning signals.
Section 4.1: CANDI (§5.3 and Appendix C.3) already observes that the transformer operates on projected noisy one-hot inputs and exploits this embedding-space view to accelerate sampling.
Section 4.2: RePlaid and LangFlow already analyze how CE and MSE shape embedding geometry, while CANDI studies token identifiability in embedding spaces.
Section 4.3: CANDI (Appendix B.3.2) explicitly interprets SC as supplying pseudo-clean anchors that address the lack of identifiable conditioning signals.
The paper also acknowledges many connections. Nevertheless, I would appreciate a clearer distinction between established observations, new analytical insights, and the modifications they motivate.
W2 | I have a concern about the practical value of the DoubleCat adaptation to Plaid. The proposed variants improve over Plaid w/o SC but remain largely below standard Plaid w/ SC. Given this accuracy gap, I am not convinced that saving the extra training forward pass offers a worthy trade-off.
Therefore, I would like to see posterior DoubleCat combined with the standard SC and compared against Plaid + SC.
W3 | SC remains largely beneficial in both one-hot CDLMs and Plaid. The results do not establish whether DoubleCat’s gains reflect partial mitigation of the same failures addressed by SC or complementary effects. It would be valuable to see more direct evidence connecting the gains to the proposed mechanism.
W4 | In Figure 5, DoubleCat does not show noticeable improvements on OWT. Thus, the SemiCat gains maybe attributed to the CANDI's revealing mechanism rather than DoubleCat, which is the key methodological contribution, not the revealing mechanism. The scaling experiments likewise do not isolate DoubleCat’s contribution.
Questions for authors
-
Which analytical insights are new relative to the cited prior work, and how do they specifically motivate the proposed modifications?
-
Does posterior DoubleCat improve over Plaid + SC when both use the same SC protocol?
-
What evidence distinguishes a shared mismatch-reduction mechanism from complementary benefits of DoubleCat and SC?
-
Could the author provide scaling results for DoubleCat alone, ideally also including the base flow with CANDI-style reveals alone?
4 comments
W4, Q4
the SemiCat gains maybe attributed to the CANDI's revealing mechanism rather than DoubleCat [...]
In our original submission, we provided the 2x2 matrix for this: CANDI is one-hot with reveals (now clarified in the background, L. 148), and SemiCat is DoubleCat+CANDI:
| GSM8K accuracy (%, best over samplers) | One-hot | DoubleCat |
|---|---|---|
| Without reveals | 4.42 (base one-hot flow) | 16.00 (DoubleCat) |
| With reveals | 9.63 (CANDI) | 32.90 (SemiCat) |
This clearly identifies that the gains from CANDI and DoubleCat are complementary, in fact super-additive. DoubleCat is on par with the one-hot flow on OWT, but is clearly superior on TinyGSM: it never hurts, and the gains are substantial for conditional generation. A scaled run is very costly (256 GPUs for about 5 days), and so we find the small scale evidence appropriate; we have nevertheless launched it. We cannot guarantee that it will finish on time, but will report results as soon as possible, and intermediate ones in the meantime. We have not launched a CANDI run.
To conclude, we are thankful for the reviewer's comments that help outline the paper's novelty. We remain available for further clarifications.
References
[1] Why Gaussian Diffusion Models Fail on Discrete Data and How to Prevent It?, Shabalin et al. (2026). [2] Flow Map Matching with Stochastic Interpolants, Boffi et al. (2026).
I sincerely thank the authors for their response, which addresses most of my concerns. I appreciate the clarification of the paper’s novelty, the new analysis of the train–inference mismatch, and the authors’ explicit acknowledgment that adding posterior DoubleCat to Plaid + SC lowers overall accuracy (Tab. 11).
My remaining concern (W4) is whether the gains from DoubleCat alone extend beyond TinyGSM. These gains on TinyGSM are clear, but DoubleCat performs similarly to the baseline on OWT, and the scaled DoubleCat results are on the way.
Overall, I find DoubleCat to be a simple and promising idea supported by a valuable analysis. Therefore, I am raising my score to 3 and look forward to seeing the scaled DoubleCat results.
We thank the reviewer for raising their score, and for a discussion that genuinely improved the paper.
My remaining concern (W4) is whether the gains from DoubleCat alone extend beyond TinyGSM. [...] the scaled DoubleCat results are on the way.
The 1.7B DoubleCat run has not finished yet, so we compare all models at the same fraction of training (57%). DoubleCat alone clearly improves generation over the one-hot flow, at every NFE: at matched entropy, its generative perplexity is 10--26% lower, and SemiCat lowers it by a further factor of ~2 (new App. G, Fig. 14).
On likelihood benchmarks, DoubleCat and the one-hot flow are on par (Table 13). This is expected: these benchmarks score the model on its training interpolant, where no train--inference mismatch arises; the mismatch DoubleCat addresses only appears at sampling time, when the model reads its own predictions.
We have added these results to the paper, and will update them with the final checkpoints.
We thank the reviewer for their valuable observations; the resulting changes have been highlighted in blue in the revised PDF.
W1, Q1
We concur, and have modified the text to make the connection more explicit (L. 205--206, 253). Our results are complementary, as [1] leverage a multimodal lens, whereas we show that the interpolant construction can be improved, without using SC. We have now highlighted the motivation given by the inference path's shape (L. 197--200), I^t=(1−γt)X0+γtY^t.
This is true. We are happy to revise any statement that might suggest otherwise.
We agree, and have merged 4.1 into a paragraph in the old 4.2. The novelty is that we leverage this to train embedded methods, instead of using that for inference (L. 377).
RePlaid analyses empirically the embedding geometry of its own model. LangFlow's analysis revolves around eq. (48), the gradients of an MSE loss, and collapse when training only with MSE. We push this further by showing how the DER of an embedded model reduces to that of a one-hot flow, and exploring the gradient of both CE and MSE objectives (App. E.3).
We concur, and have changed the abstract, introduction and conclusion to better reflect our main contributions, as reviewer yuNT suggested too, with a clearer focus on DoubleCat, its scaling, and the original section 4.2 (embeddings reducing to one-hot DER) (L. 19--22, 54--58, 498--499).
How insights motivate methods. (i) Inference visits states rare at training (prior: [1]); as Y^t stays off the vertices, DoubleCat trains on such states (ours). (ii) Token identifiability (prior: CANDI) motivates the posterior as SC, and SemiCat's reveals. (iii) One-hot flows are embedded flows (prior: CANDI); we use this to train embedded DoubleCat/SemiCat, and our DER reduction (ours) explains why CE-trained embeddings recover one-hot DERs (§3.1, §3.2, §4.1, §4.2).
Several other explorations are included in App. D and F.
W2, Q2
We agree that the SC results are better. The goal was to gain understanding on what SC provides, and the token-wise posterior seems to recover part of the performance. While SC remains stronger, it becomes costly when accelerating a model (e.g., a flow map [2] requires 3 forward passes per training step, up to 6 with SC), hence our willingness to bypass it. We add this motivation (L. 167--169).
Under the same protocol (pinned endpoints, SC), adding posterior DoubleCat gives 29.64% vs 32.98%; at the default decode it is slightly ahead (26.69% vs 26.08%), but SC alone benefits more from decode tuning. Therefore, the two are not complementary, consistent with addressing the same mismatch. We have included those numbers in App. E.6 (Table 11).
W3, Q3
We provide new evidence for this. Using OWT trained checkpoints for one-hot flow and DoubleCat, we gathered one-hot and DoubleCat inference trajectories (I^toh, and I^tdc, resp.), using 128 NFE Euler integration. We compare the distances between one-hot flow inference and DoubleCat interpolants (Itdc); DoubleCat inference and DoubleCat interpolants; one-hot flow interpolants (Itoh) and one-hot flow inference. We use the conditional and marginal Wasserstein distances, and MMD. (Details in App. D.7, Figure 8.)
We can see that, for times up to t≈0.75 or 0.8 (depending on the distance), the train--test gap is lower for DoubleCat, with large advantages until around t=0.5. Near t=1, the DoubleCat interpolant's marginals come close to those of one-hot flows by construction, and so the advantage narrows there, and flips slightly in favour of the one-hot flow. Crucially, the one-hot flow inference is closer to DoubleCat's interpolants than it is to its own under all three distances, for all times, although the margin narrows again near t=1.